Module 6 of 10 · Beginner track
Model selection — which variables deserve to stay?
Real datasets come with more candidate predictors than useful ones. Which belong in the model? The brute-force answer — try every subset — explodes combinatorially (8 features is 256 models; 30 is a billion), so the classical workhorse is forward stepwise selection: start empty, repeatedly add the one feature that improves the fit most, and produce a ranked path of models from 1 feature to all of them.
The path is the easy part. The hard part is where to stop, because the obvious scorecard is rigged: adding anyfeature — including pure noise — can only lower training RSS, so train R² marches upward forever, always whispering “one more.” You need a judge that charges rent per feature: adjusted R²(a small per-feature penalty; AIC and BIC are its more principled relatives), or better, the estimated test error from Module 5's resampling. The lab plants 3 real features among 5 decoys and lets you watch all the scorecards disagree.
🎛 Stepwise selection lab
Order chosen by forward stepwise (3 features are real, 5 are decoys)
Adjusted R² picks
4 features
True test MSE picks
4 features
R² vs adjusted R²
True test MSE
Forward stepwise adds whichever remaining feature most improves the training fit. The real features go first (green badges) — then it keeps going, happily adding decoys, and train R² keeps inching up as it must. Adjusted R² charges a small rent per feature and peaks near the truth; the red test-MSE curve renders the final verdict. With small n and big noise, watch a decoy occasionally sneak in before the third real feature — selection is noisy, which is why Module 5's resampling (or the lasso from the advanced track) should confirm it. Educational tool.
From discrete choices to a smooth dial
Stepwise selection is a sequence of hard in-or-out verdicts, and hard verdicts made on noisy evidence are themselves noisy — rerun on a fresh sample and the chosen set changes. The modern refinement replaces the discrete search with a continuous one: keep everyfeature but penalize coefficient size, letting λ shrink the useless ones smoothly to zero. That's the ridge/lasso story, told hands-on in the advanced track's shrinkage module— same goal as this page, better statistical manners. Either way, the meta-rule from Module 5 stands: the selection procedure itself must face resampled judgment, or the winning subset's score is as inflated as any other training-set number.
Things to try
- • At default settings, read the badge order: the three real features almost always go 1-2-3 (in effect-size order), then the decoys trickle in. Selection works — up to a point.
- • Raise noise to 2.5 with n = 20 and re-read the badges: decoys start jumping the queue. Scarce, noisy data breaks selection before it breaks fitting.
- • Compare where the green (adjusted R²) and red (test MSE) verticals land — usually near 3, sometimes disagreeing by one. Model selection has error bars too.