Module 2 of 10 · Beginner track
The bias-variance tradeoff, seen directly
Module 1 showed that flexible models can fool you. This module explains why, with the most useful equation in the subject. A model's expected error on new data breaks into exactly three parts:
test error = bias² + variance + irreducible noise
- Bias — error from rigidity. A straight-line model fitted to a curved truth is wrong on average, no matter how much data you give it.
- Variance — error from sensitivity. A flexible model gives a different answer every time the data is redrawn; on any one draw it's partly fitting that draw's luck.
- Noise — the floor. The part of y that nothing observable explains; no model goes below it.
Here's the catch: in real life you get one dataset, so you can never see bias and variance separately — you only feel their sum. The lab below cheats on your behalf: it draws 30 parallel-universe training sets from the same truth, refits your chosen model on each, and shows you both the spaghetti of fits and the decomposition it implies.
🎛 Parallel-universe lab
Bias²
0.135
Variance
0.099
Noise σ²
0.250
Expected test MSE
0.483
12 refits on different training sets (degree 3)
The decomposition vs flexibility
Left: the same model, refit on 12 parallel-universe training sets. At low degree the grey curves agree with each other but miss the green truth (bias); at high degree they hug the truth on average but disagree wildly with each other (variance). Right: averaging over many universes turns that picture into the famous U — blue falls, amber rises, and the red total bottoms out where they trade evenly. Normally this decomposition is invisible (you get oneuniverse); simulation is the only place you can watch it. Educational tool.
How to read the two pictures
Left panel, low degree: the grey fits agree with each other (low variance) but all miss the green truth the same way (high bias) — consistently wrong. High degree: they track the truth on average but disagree wildly (high variance) — unreliably right. The right panel averages this over many universes: bias² falls as flexibility grows, variance rises, and their sum — the red curve — makes the U whose bottom is the model you actually want. Every tuning knob in machine learning (polynomial degree, k in kNN, tree depth, regularization λ) is a lever on this same tradeoff.
Things to try
- • Degree 0: the fits are nearly identical flat lines — total agreement, total wrongness. Degree 10: chaos that averages out right. Watch the tiles flip dominance.
- • Double the sample size and watch the variance curve drop while bias² doesn't move — more data buys you the right to more flexibility, but never fixes rigidity.
- • Raise σ: the whole red curve lifts (the floor rose) and its minimum shifts left — noisier data wants simpler models.