Module 6 · Linear models
Kernel smoothing — no model, just neighbors with weights
Module 5 built flexibility by choosing clever features. Kernel smoothing takes the opposite route: fit nothing in advance. To predict at a point x₀, take a weighted average of the training responses, with weights that fade smoothly with distance from x₀ — the fading profile is the kernel, and its width is the bandwidth h. It's k-nearest-neighbors with the rough edges sanded off: instead of an abrupt in-or-out vote, everyone votes with a weight.
One number now controls everything. Small h: only immediate neighbors matter — the fit hugs every noisy wobble (variance). Large h: the average reaches far — real bumps in the signal get ironed flat (bias). The bandwidth is the bias–variance dial of Module 2, in its purest form.
There's a subtle failure worth seeing with your own eyes: at the edges of the data, the averaging window is one-sided — at the left boundary, every neighbor lies to the right — so if the signal slopes there, the plain average is pulled systematically inward. Local linear regression fixes this by fitting a tiny weighted line(not a constant) inside each window; the line can tilt with the trend, and the boundary bias cancels to first order.
🎛 Kernel smoother lab
Test MSE
0.153
f̂(x₀) = 0.32
The amber bump at the bottom is the kernel: how much each training point counts when predicting at x₀. Small h → narrow bump, wiggly fit (variance); big h → wide bump, oversmoothed fit (bias). Drag x₀ to the edge: the plain weighted average bends toward the interior because its window is one-sided there — boundary bias — while local linear regression fits a tilted line inside the window and corrects it automatically. Educational tool.
How the lab computes it
weight: wᵢ = K((xᵢ − x₀)/h), K = Gaussian bump NW: f̂(x₀) = Σ wᵢyᵢ / Σ wᵢ (weighted average) local linear: fit (a,b) minimizing Σ wᵢ(yᵢ − a − b(xᵢ−x₀))², f̂(x₀) = a
The price of memory-based methods: there's no compact fitted object — prediction time scales with the training set, and in high dimensions the kernel neighborhood suffers the same curse as kNN. The same “local weighting” idea extends to local likelihood (a logistic fit in every window) and to kernel density estimation, where the smoothed thing is the data distribution itself.
Things to try
- • Sweep h from 0.05 to 1.5 and watch the fit morph from connect-the-dots to a flat mush. The test MSE tile bottoms out in between.
- • Drag x₀ to −3 on the weighted-average tab: the amber bump gets chopped off by the boundary and the fit visibly detaches from the green truth. Switch to local linear — the detachment heals.
- • Note that changing x₀ never changes the black curve on the interior — it just moves the spotlight showing which points would vote there.