double descent

Apr 17 2025

Off with the polynomial’s tail!

Alex Shtoff

Intro Last time we did a small curve fitting exercise - we fit high degree polynomials represented in the Legendre basis to a curve, and observe the celebrated “double descent” phenomenon: after crossing the memorization threshold, the generalization error improves as the degree increases. Then,...

machine learning model pruning scikit-learn double descent feature selection polynomials polynomial regression california housing dataset feature engineering legendre polynomials

Mar 27 2025

Let the polynomial monster free

Alex Shtoff

Intro In a recent post by Ben Recht, titled Though Shalt Not Overfit, Ben claims that overfitting in the way that it is colloquially described in data science and machine learning, doesn’t exist. Indeed, there is the famous double descent: trained neural networks that have much more parameters...

machine learning generalization double descent fourier features polynomials overparameterization polynomial regression chebyshev polynomials feature engineering legendre polynomials