Module 5 of 10 · Beginner track
Resampling — four estimates of one number
By now the rule is drilled in: judge a model on data it didn't train on. The engineering question is how to arrange that when data is finite, and the standard answers form a ladder:
- Validation split — hold out (say) half, once. Simple, but you wasted half the data for fitting, and your estimate depends on which half fate handed you.
- K-fold CV — split into K parts; each part takes one turn as the judge while the rest train. Every point gets judged exactly once, and the K scores average into something much steadier.
- Leave-one-out (LOOCV) — K = n: each single point takes a turn. No split-luck at all, at the price of n refits.
The lab makes the ladder's payoff visible in a way single runs never do: it repeats each random-split method 15 times on the sametraining data and plots all the answers as dots, next to the true test error (computed on 2,000 secretly held points). The question isn't which method lands closest once — it's which method you could trust having run it only once, which is all you ever get.
🎛 Estimate-spread lab
Half-split spread
±0.089
mean 0.358
5-fold spread
±0.018
mean 0.388
10-fold spread
±0.011
mean 0.383
LOOCV
0.373
deterministic — one answer
All four columns estimate the samenumber — this model's true test error (red dashed, computed on 2,000 hidden points) — from the sametraining set; the dots vary only by the luck of the random split. The half-split column scatters widest (one arbitrary split, half the data wasted); K-fold tightens it by averaging K held-out scores; LOOCV removes split-luck entirely. Costs: LOOCV means n refits, and half-split's smaller fitting set makes it slightly pessimistic — check where each column's cloud sits relative to the red line. Educational tool.
Reading the columns like a statistician
Two separate defects are on display. Spread (variance of the estimate): the half-split cloud is tallest because one arbitrary coin flip decides which points judge and which teach. Position (bias of the estimate): the half-split cloud also tends to sit a touch above the red truth, because its models trained on only half the data and are genuinely a bit worse — a pessimism that shrinks as the fitting fraction grows through 5-fold, 10-fold, LOOCV. 10-fold is the everyday sweet spot. For time-series and trading data one more rule is non-negotiable: folds must respect time order (train on past, judge on future) — shuffled CV quietly leaks tomorrow into yesterday.
Things to try
- • Shrink n to 30: every cloud fattens, and the half-split becomes almost uninformative — resampling matters most exactly when data is scarce.
- • Crank the model degree to 8: the true error jumps (overfitting), and all four methods report it — honest estimation doesn't require knowing the truth, just discipline about who judges.
- • Compare the 5-fold and 10-fold clouds: the improvement from 5→10 is real but modest — the big win was leaving the single split behind.