Module 7 · Foundations
Model assessment — grading without cheating
Every module so far ended with the same cliff-hanger: the best flexibility “would be found by a validation set.” This module is that machinery. The problem it solves: training error is optimistically biased — the model was chosen specifically to score well on those points, so its score on them overstates how good it is. The more flexible the model, the bigger the lie.
With unlimited data you'd carve out a big untouched test set. With limited data, K-fold cross-validation recycles what you have: split the training set into K folds, and K times over, fit on K−1 folds and score on the held-out one. Every point gets scored exactly once by a model that never saw it, and the average of those scores estimates true test error — using nothing but training data.
The lab runs this live: for each candidate flexibility it shows training error (blue, the liar), the K-fold CV estimate (amber, the honest broker) and — because this is synthetic data — the true test error (red) that in real life you never observe. The question is how well amber tracks red.
🎛 Cross-validation lab
CV picks degree
8
CV MSE 0.170
1-SE rule picks
8
simplest within one SE
True-best degree
3
from the 400-pt test set
Blue keeps falling forever — training error is a rigged exam. The amber CV curve, computed only from the training data by holding out each fold in turn, tracks the red truth (which normally you never get to see) remarkably well. The circle marks the one-standard-error rule: take the simplest model whose CV score is within one SE of the minimum — a principled nudge toward parsimony. Educational tool.
Choosing K, and the one-standard-error rule
K trades bias for variance and compute: K=N (leave-one-out) trains on almost all the data (low bias) but its folds overlap heavily; K=5 or 10 is the practical default. Since the CV curve is itself a noisy estimate, chasing its exact minimum overfits the selection — the one-standard-error rule says: take the simplest model whose CV score is within one standard error of the best. When in doubt, simpler.
The classic silent failure is leakage: any step that looked at all the data — picking features by correlation with the outcome, scaling by the full dataset's statistics — must happen insideeach fold, on that fold's training part only. Screen features on everything first and CV will happily bless a model built on noise; in a p ≫ N setting (Module 18) this single mistake can manufacture near-perfect fake accuracy.
Things to try
- • Compare K=2 with K=10: with only half the data per fit, K=2's amber curve sits pessimistically above the red truth at high flexibility.
- • Shrink n to 20 and re-read the picks: with scarce data the CV curve gets noisy and the 1-SE rule's conservatism starts earning its keep.
- • Run the comparison a few times across noise settings: CV's chosen degree hovers around the true-best degree but doesn't always hit it — honest estimation, not clairvoyance.