Module 15 · Trees & ensembles
Random forests — wisdom of a deliberately diverse crowd
Module 9ended with the tree's fatal flaw: instability. A deep tree is low-bias but high-variance — jiggle the data, get a different staircase. Module 8supplied the cure: if you can get many independent-ish versions of a noisy estimate, average them and the noise cancels. A random forest is that cure, industrialized.
Averaging identical trees would cancel nothing — the trick is manufacturing diversity, twice over. First, each tree trains on its own bootstrap sampleof the rows (bagging). Second — the forest's own innovation — at every split, the tree may only consider a random subset of the features(√p of them is the classification default). Without that second trick, one dominant feature would headline every tree's first split and the trees would err in unison; forcing variety decorrelates their mistakes, and decorrelated mistakes are the ones averaging destroys.
The variance algebra says it plainly: averaging B trees with pairwise correlation ρ leaves variance ρσ² + (1−ρ)σ²/B. The second term dies as B grows; the first is the floor set by correlation. More trees never hurt — the floor is what feature-subsampling lowers.
🎛 Forest lab
Single deep tree · test MSE
0.316
jagged and overfit
Forest of 20 · test MSE
0.242
same trees, averaged
Improvement
23%
variance averaged away
One tree vs the forest
Test error vs number of trees
Every tree here is deep and overfit on purpose — low bias, high variance — but each is grown on a different random subsample, so their mistakes disagree. Averaging keeps the shared signal and cancels the disagreements: the amber staircase becomes the calm black curve. The error curve flattens rather than turning up — more trees never overfit; they only stop helping. (Full random forests add a second decorrelation trick: each split considers only a random subset of features.) Contrast with boosting, where trees are sequential and more rounds eventually do overfit. Educational tool.
Two free gifts of the bootstrap structure
Out-of-bag (OOB) error.Each bootstrap sample leaves out ~37% of the rows; those rows are a ready-made test set for that tree. Score every point using only the trees that didn't train on it and you get an honest error estimate with zero extra computation — cross-validation included in the price.
Variable importance.Shuffle one feature's values in the OOB data and measure how much accuracy drops — repeat per feature. Rough but battle-tested; one caveat worth knowing: correlated features share (and thus dilute) each other's importance. Between OOB, importances and near-zero tuning burden, the forest is the strongest “fit-it-first” baseline in tabular ML — the number a fancier model has to beat.
Things to try
- • Slide trees from 1 to 60 and watch the amber staircase melt into the calm black curve — same overfit ingredients, radically better dish.
- • Set subsample fraction to 1.0: every tree sees nearly the same data, diversity collapses, and the improvement shrinks. Diversity is the active ingredient, not tree count.
- • Read the error-vs-trees curve: it flattens and stays flat. Compare boosting's curve in Module 10, which eventually turns up — the deep difference between parallel averaging and sequential correction.