Module 3 · Linear models
Linear regression — and the case for shrinkage
Least squares finds the weights that minimize squared error, and among unbiasedlinear estimators it's the best you can do. The catch is in the word unbiased. With many predictors — especially correlatedones — the unbiased solution has enormous variance: the data can't tell nearly-redundant features apart, so their coefficients take huge offsetting values that swing wildly from sample to sample. Unbiased, and useless.
Shrinkage is the deliberate trade: accept a little bias to buy a big variance reduction. Add a penalty on coefficient size to the least-squares objective, and the whole character of the fit changes with which penalty you pick:
- Ridge penalizes λ·Σβ² (L2) — every coefficient is pulled smoothly toward zero, correlated features share credit gracefully, but nothing is ever removed.
- Lasso penalizes λ·Σ|β| (L1) — the corners of the absolute-value penalty pin coefficients at exactly zero, so raising λ performs automatic variable selection.
The lab fits both to a dataset where only 3 of 6 features are real and the features share a common factor. Watch the coefficient paths as the penalty λ grows.
🎛 Ridge / lasso path lab
Test MSE
2.92
Nonzero coefficients
5 / 6
truth: 3 real, 3 useless
Coefficients
1.6 -1.4 1.3 0.1 0.2 0.0
truth: 2.0 −1.5 0.8 0 0 0
Coefficient paths vs log₁₀ λ
Test MSE vs log₁₀ λ
Solid paths are the three real coefficients, dashed are the three useless ones. Ridge squeezes every path smoothly toward zero but never to zero; lasso kills the useless ones exactly — shrinkage and variable selection in one penalty. The test-MSE dip on the right shows the sweet spot a validation set would find. Educational tool.
Why L1 selects and L2 doesn't
Picture the constraint region each penalty allows: ridge's is a sphere, lasso's a diamond. The least-squares contours expand until they first touch the region — a sphere gets touched at a generic point (all coordinates nonzero), but a diamond gets touched at a corner, where some coordinates are exactly zero. Same objective, different geometry, completely different behavior. Elastic net blends the two penalties to get selection and graceful handling of correlated groups.
ridge: β̂ = (XᵀX + λI)⁻¹ Xᵀy (closed form, always exists) lasso: no closed form — coordinate descent, soft-thresholding each βⱼ
Things to try
- • On lasso, raise λ and watch the three dashed (useless) paths hit zero first, then stay there — the real ones survive longest. That ordering is the selection.
- • Switch to ridge at the same λ: the useless paths get small but never exactly zero.
- • Set feature correlation to 0.9 and watch the low-λ ends of the paths go wild — that instability is exactly the variance shrinkage exists to kill.
- • Find the λ minimizing test MSE and note it's well above zero: plain least squares is never the best point on the path here.