Module 18 · Neighbors & structure
When features outnumber observations
Genomics hands you 20,000 gene expressions for 100 patients. A quant screen hands you 5,000 candidate signals for 500 trading days. This is the p ≫ Nworld, and classical fitting doesn't merely struggle there — it fails with mathematical certainty. Once p ≥ N, least squares can fit the training data exactly, whatever it is: shuffle the outcomes into nonsense and you'll still get a perfect in-sample fit. A perfect fit that would fit anything is evidence of nothing.
The lab makes the collapse visible: 50 observations, 4 real features, and a slider that piles on useless ones. Watch what each extra noise feature does to least squares as p climbs toward N — and what a ridge penalty (Module 3) does about it.
🎛 p ≫ N lab
Least squares · test MSE (p=20)
1.59
Ridge · test MSE
1.35
Irreducible noise σ²
1.00
the best anyone can do
Only 4 of the p features carry signal; every extra one is pure noise, yet each hands least squares another free parameter. As p approaches N = 50 (black dashed line) the red curve explodes — with as many parameters as data points, least squares fits the training set exactly, noise and all, and its predictions become garbage. Ridge's bet is that heavy shrinkage plus a little bias beats unbiased-but-insane: the green curve degrades gracefully and stays near the noise floor (grey dashed). In the p ≫ N world — genomics, text, image features — some form of this bet isn't optional. Educational tool.
The rules of survival in high dimensions
- Regularize — it's not optional. Ridge, lasso, elastic net: some prior belief (coefficients are small; most are zero) must substitute for the data you don't have. The bet that bias beats variance, made mandatory.
- Fear multiplicity. Screen 5,000 noise features for correlation with the outcome and the best few will look spectacular by chance alone. Selection must be validated inside the resampling loop (Module 7's leakage warning, now at maximum stakes) — and false-discovery-rate control, not per-test p-values, is the right yardstick for the screen itself.
- Prefer structure. Factor models, sparsity, graphs (Module 17): each is a way of saying “the world has fewer degrees of freedom than my feature count,” which is the only reason p ≫ N problems are solvable at all.
For anyone doing quantitative trading research this module is the whole sermon: backtesting many signals on limited history is a p ≫ N screen, and the best backtest of a large batch is, by construction, the luckiest.
Things to try
- • Walk p up from 4 toward 50 and watch the red curve leave the building as it crosses N. The green ridge curve barely notices.
- • Push p to 150 — three times more parameters than data points — and note ridge still sits near the noise floor. Shrinkage is doing all of the work.
- • Now weaken ridge (log λ = −2): with tiny regularization it inherits least squares' disease. The penalty strength isn't a detail; it's the load-bearing wall.