Module 16 · Trees & ensembles
Ensembles — hiring models that disagree
Random forests average many versions of one model type. The broader ensemble idea drops that restriction: combine differentmethods — a linear model, a spline, a kernel smoother, a boosted tree — and let their disagreements cancel. The condition for a blend to beat its best member is not that the members be good; it's that their errors be imperfectly correlated. A model that's wrong in a new waycan be worth more to the committee than a slightly better model that's wrong in the same old way.
So how do you choose the weights? By hand, if you enjoy that (the lab lets you try). The principled answer is stacking: treat each base model's prediction as a feature, and regress the true outcomes on those features — with the crucial caveat that both the predictions and the outcomes must come from data the base models didn't train on. Fit weights on training-set predictions and the most overfit base model — the one that memorized best — grabs all the weight, exactly wrong.
🎛 Blending lab
Linear · test MSE
0.726
Spline · test MSE
0.256
Kernel · test MSE
0.211
Blend · test MSE
0.285
worse than the best base
The three pale curves fail differently: the line is too stiff, the narrow kernel too jumpy, the spline in between. Because their errors are imperfectly correlated, a weighted average can beat all three — try to find weights by hand, then press the button: stackingfits the weights by regressing held-out outcomes on the models' held-out predictions (held out, or the wiggliest model would grab all the weight for having memorized the training set). Educational tool.
The ensemble family tree
- Bagging / forests (Module 15): same learner, resampled data, equal weights. Attacks variance.
- Boosting (Module 10): same learner, sequential, each member trained on the errors so far. Attacks bias.
- Stacking: different learners, weights learned on held-out predictions. Attacks the fact that no single method family is right for every dataset.
- Bayesian model averaging: weight models by how probable each is given the data — the fully probabilistic cousin, at home when the model list is small and interpretable.
The limits are as instructive as the wins: ensembling identical-ish models buys nothing, blending destroys interpretability, and a stacked blend of leaky base models launders the leak into “validated” performance. Kaggle-style stacks of stacks win contests by hundredths of a point; in production, one boosted model plus one honest validation scheme is usually the better trade.
Things to try
- • Zero out two models and check each base model's solo test MSE, then hunt by hand for weights beating the best. It's findable — and fiddly.
- • Press the stacking button and compare its weights to your hand-tuned ones. Note it may assign a small weight to the worst model — even a bad model contributes if it errs differently.
- • Look at where each pale curve fails (edges for the kernel, bends for the line) and where the black blend fails — the blend's errors are smaller and more uniform. That's decorrelation at work.