Module 8 · Foundations
The bootstrap — uncertainty by brute force
You fit something to data — a regression slope, a median, a Sharpe ratio, a full model. How much would that number change if you could rewind and collect a fresh dataset? For a textbook estimator under textbook assumptions there's a formula. For everything else, there's the bootstrap: treat your sample as a stand-in for the population, draw new samples from it — same size, with replacement — and refit on each. The spread of the refitted estimates is your uncertainty, measured rather than assumed.
It feels like cheating and isn't: the empirical distribution of your data is a legitimate (and, past modest sample sizes, good) estimate of the true one, and resampling from it simulates exactly the experiment you couldn't run. The lab bootstraps a regression slope on data with deliberately heavy-tailed noise — a case where the textbook Gaussian formula is exactly what you shouldn't trust.
🎛 Bootstrap lab
Observed slope β̂
1.105
true value: 0.800
Bootstrap SE
0.288
std-dev of resampled β̂*
95% interval
[0.52, 1.63]
percentiles of β̂*
Each bar counts bootstrap resamples (rows drawn with replacement from the one observed dataset) whose refitted slope landed there. The spread of this histogram is the sampling uncertainty of β̂ — no formula, no normality assumption needed, which matters here because the noise has heavy tails. Black line: the observed estimate. Green dashed: the true 0.8. Shrink n or crank the noise and watch the histogram fatten. Educational tool.
From measuring wobble to removing it: bagging
Read the histogram again with a builder's eye. Each bootstrap refit is a slightly different, equally plausible model. If instead of studying their spread you average their predictions, the wobble partially cancels — that's bagging (bootstrap aggregating). It does nothing for a stable estimator like this slope, but for unstable, jumpy fits — decision trees above all — averaging away the variance is transformative. Module 15 builds an entire method on exactly this observation.
The bootstrap also quietly powers honest ML practice: confidence bands on feature importances, stability checks on selected variables (does the lasso pick the same features on every resample?), and out-of-bag evaluation — each resample leaves out about a third of the points, a free validation set.
Things to try
- • Push B from 50 to 1000: the histogram's shape stabilizes — B controls simulation smoothness, not the underlying uncertainty. n and noise control that.
- • Set n to 10: the interval balloons and goes lopsided. The bootstrap reports asymmetry a symmetric ±2·SE formula would hide.
- • Check whether the green line (truth, 0.8) falls inside the 95% interval — usually yes, occasionally no. That's what 95% means.