Module 1 · Foundations
What is statistical learning?
Statistical learning is the discipline of extracting a usable rule from data. You observe inputs and outcomes — borrower characteristics and whether they defaulted, yesterday's positioning and today's return, a pixel grid and the digit it shows — and you want a function that predicts the outcome for cases you haven't seen.
Two broad settings run through this whole curriculum. In supervised learning every training example comes with the right answer attached — a label to predict (classification) or a number (regression). In unsupervised learning (Module 14) there are no answers, only structure to find: clusters, low-dimensional directions, dependency graphs.
The central tension — the one every later module elaborates — fits in one sentence: a model must be flexible enough to capture the signal, but every drop of flexibility beyond that gets spent memorizing noise. And you cannot tell the difference by looking at performance on the data you trained on. Try it yourself below: the dial is model flexibility, and the two numbers to watch are the error on the training points versus the error on 200 fresh points the model never saw.
🎛 Overfitting playground
Training error (MSE)
0.212
on the points it saw
Test error (MSE)
0.275
on 200 fresh points
Generalization gap
0.064
test − train
The green dashed line is the true signal, the dots are noisy observations of it, and the black line is what the model learned. Degree 0–1 can't follow the signal (underfit); degree 12+ chases individual noise dots (overfit) — training error keeps falling while test error turns back up. Educational tool.
The vocabulary that carries through all 18 modules
- Training vs test error. How wrong on data you fit vs data you didn't. The gap between them is what “overfitting” means, quantitatively.
- Signal vs noise. The data is truth plus randomness: y = f(x) + ε. Even a perfect model carries the irreducible error of ε.
- Flexibility / complexity. How many effective knobs the model has. Degree-15 polynomial, k=1 nearest neighbor, depth-10 tree — different costumes, same danger.
- Generalization. The only thing that counts: performance on cases the model hasn't seen. Everything in Module 7 is about estimating it honestly.
Things to try
- • Set degree to 1 — both errors are high and equal: underfitting. Set it to 15 — training error collapses, test error explodes: overfitting. Somewhere near 4–6 the test error bottoms out.
- • Drop the training points to 10 and watch how much less flexibility you can afford. More data buys the right to more complexity.
- • Crank the noise. The best achievable test error rises with it — no model, however clever, beats the irreducible ε.