Module 8 of 10 · Beginner track
Trees — watching the plane get carved
A decision tree classifies by interrogation: is x ≤ 1.3? If yes, is y ≤ −0.4? — each answer narrowing down to a final verdict. Because every question involves a single feature, the geometric footprint is distinctive: the feature space gets diced into axis-aligned rectangles, one per leaf, each voting its local majority. In two dimensions you can watch the whole thing happen — which is exactly what the lab below does.
Splits are chosen greedily to make the resulting groups as pureas possible, measured by Gini impurity (a purity score that's zero when a region is all one class). The rectangle picture makes the tree's personality obvious: any boundary that runs diagonally gets approximated by stair-steps, deep trees carve private boxes around single noisy points, and small changes in the data can redirect an early split and reshape the entire map — the instability that makes single trees weak predictors despite their charm.
🎛 2D partition lab
Leaf regions
4
axis-aligned rectangles
Training error
12.9%
Test error
8.8%
on 400 fresh points
Each rectangle is a leaf; its label inside is the share of class-1 (amber) training points it holds, and its fill is the class it predicts. Every split is a single question about one feature — “x ≤ 1.3?” — chosen to minimize Gini impurity, so the boundaries are always vertical or horizontal. Depth 0 is one region (predict the majority); each level potentially doubles the rectangles. Watch the tree approximate the diagonal class boundary with stair-steps — the price of axis-aligned questions — and at high depth, tiny rectangles forming around individual points. Educational tool.
From one tree to a crowd of them
The chapter's punchline is that the tree's weakness — high variance — has a known cure: grow many trees and combine them. Bagging trains each on a bootstrap resample and averages; random forests add feature-subsampling at every split so the trees disagree in more useful ways; boosting instead grows small trees sequentially, each correcting the last. All three are built hands-on in the advanced track (forests, boosting) — and between them they turn this module's wobbly rectangles into the strongest default learners for tabular data.
Things to try
- • Step depth 0 → 1 → 2 and watch the first cut land where it best separates the colors, then each half get its own best cut. Recursive greed, visualized.
- • Depth 5, min-leaf 2: find the tiny rectangles enclosing one or two points — memorization you can point at. Raise min-leaf and watch them merge away.
- • Look at the percentage labels: leaves near 50% are the tree admitting uncertainty. A good pruning would merge those first.