Module 5 · Linear models
Basis expansions — flexibility without leaving linear-land
Here's a trick that powers half of applied statistics: a “linear” model only has to be linear in its coefficients. Replace the raw input x with a set of transformed features — h₁(x), h₂(x), … — and least squares happily fits curves, while all the linear theory (closed forms, standard errors, shrinkage from Module 3) still applies. The whole question becomes: which basis?
The obvious answer — powers of x — is a trap. A degree-10 polynomial is one global formula: every coefficient affects the fit everywhere, so bending it to fit data in the middle makes it thrash at the edges, and just beyond the data it explodes. The professional answer is splines: chop the axis at knots, fit a low-degree polynomial in each segment, and force the pieces to join smoothly (equal value, slope and curvature at each knot). Flexibility now comes from adding knots — a local operation — instead of raising the global degree.
A natural cubic spline adds one more constraint born from humility: beyond the outermost knots, where there is no data to say otherwise, the fit is forced to be linear. The lab's chart deliberately extends past the data so you can see what that buys.
🎛 Polynomial vs spline lab
Train MSE
0.202
Test MSE
0.264
The chart deliberately extends past the data range (grey dashed lines at ±3). A high-degree polynomial is a single global formula — wiggle it anywhere and it lashes back at the boundaries, exploding just outside the data. A natural cubic spline is local (each knot region has its own cubic, joined smoothly) and is constrained to be linear beyond the boundary knots — the tails stay tame at the same flexibility. Light grey verticals mark the knots. Educational tool.
From knots to smoothing splines
Choosing knot counts and positions is a nuisance, and there's an elegant way out: put a knot at every data point — maximal flexibility — and control wiggliness with a penalty instead:
minimize Σ (yᵢ − f(xᵢ))² + λ ∫ f″(x)² dx
The minimizer is automatically a natural cubic spline, and λ plays exactly the role the degree/knot sliders play here — one smooth dial from interpolation (λ→0) to a straight line (λ→∞), tunable by cross-validation (Module 7). The same penalty idea reappears as “weight decay” in neural networks.
Things to try
- • Set both tabs to complexity 10 and compare the region outside ±3: the polynomial dives off the chart; the natural spline exits on a straight line.
- • At equal test MSE, count what you spent: the spline usually gets there with a calmer curve — flexibility where the data is, none where it isn't.
- • Crank noise to 0.9: both need less complexity now. The optimal basis size shrinks as data quality falls.