Module 10 · Trees & ensembles
Boosting — a committee that learns from its mistakes
Take the weakest useful model imaginable: a stump— a tree with one split, capable only of “left half predicts a, right half predicts b.” Boosting's claim is that a long sequence of stumps, each trained on what the previous ones got wrong, becomes a superb predictor. The recipe for squared error:
- Compute the current residuals — actual minus the ensemble's prediction so far.
- Fit a stump to those residuals.
- Add ν × that stump to the ensemble (ν is the learning rate, e.g. 0.1), and repeat.
The general view — the one that makes it “gradient” boosting — is that residuals are the negative gradient of the loss with respect to the predictions. Swap in a different loss (absolute error, logistic for classification, quantile) and the same machine minimizes it, one small functional gradient step per round. This is the engine inside XGBoost and LightGBM, and it remains the strongest general-purpose learner for tabular data.
Contrast with Module 15's forests, where trees are grown independently, in parallel, then averaged: bagging attacks variance. Boosting's stumps are sequential and cooperative: it attacks bias, building complexity a sliver at a time — which also means, unlike a forest, it eventually can overfit if you let it run.
🎛 Boosting lab
Train MSE @ 30 rounds
0.126
Test MSE @ 30 rounds
0.243
Ensemble fit after 30 stumps
Error vs rounds (log x)
Each round fits a one-split stump to the current residuals and adds ν times it to the ensemble. One stump is a crude step; a few dozen, each correcting the last, assemble into a smooth-ish curve. Small ν needs more rounds but generalizes better (drop ν to 0.05 and compare the red curves) — and unlike a single deep tree, test error degrades slowly when you overshoot the optimal number of rounds. Educational tool.
The slow-learning principle
The learning rate ν and the number of rounds M trade off directly: halving ν roughly doubles the M you need, but the slower-built ensemble usually generalizes better — many tiny corrections average over the noise in a way a few big ones can't. Practical GBM tuning is mostly: set ν small (0.01–0.1), pick tree depth to match the interaction order you believe in (stumps = purely additive effects, depth 2–3 = pairwise interactions), and choose M by cross-validation or early stopping on a validation set. Note the shape of the test curve in the lab: past its minimum it drifts up gently— one of boosting's most forgiving properties.
Things to try
- • Drag rounds from 1 upward and watch the fit assemble: the first stump captures the crudest high/low structure, round 30 has real shape, round 100 is polishing.
- • Set ν = 1.0: fast, greedy, and the test curve turns up early and hard. Now ν = 0.05: slower descent, later and shallower minimum.
- • Crank rounds to 400 at moderate ν and read the two error curves: train grinds toward zero while test rises only mildly — compare that to the cliff a single deep tree fell off in Module 9.