Module 9 of 10 · Beginner track
Kernels — teaching a straight-line method to draw circles
The lab below poses a problem no straight line can touch: one class sits in a central blob, the other surrounds it in a ring. Every linear classifier from Module 4 scores roughly coin-flip here — not for lack of data, but because the shapeof the answer isn't in its vocabulary.
The kernel idea expands the vocabulary without abandoning the machinery. Instead of describing a point by its raw coordinates, describe it by its similarity to every training point, where similarity fades with distance: K(a, b) = exp(−γ·distance²) — the radial kernel. A method that is linear in thosefeatures can be wildly non-linear in the original space: a circle is just “similar to the blob points, dissimilar to the ring points.” Equivalently (this is the famous kernel trick), it's as if the data were lifted into a vastly higher-dimensional space where a flat separator suffices — without ever computing that space.
Two dials govern everything. γ sets how fast similarity fades — the vocabulary's locality. The regularization sets how hard the fit may lean on individual points. Their product space contains both beautiful circles and pathological islands; the lab lets you visit both.
🎛 Radial-kernel lab
Training error
7.5%
Test error
8.5%
A blob inside a ring — no straight line can ever separate these. The radial kernel measures similarity as exp(−γ·distance²), so the “linear” machine operating on those similarities draws a circle without being told circles exist. Small γ: wide, gentle similarity — smooth boundary. Large γ with tiny λ: each point is similar only to itself, and islands form around individual points — the overfit regime (watch train error hit zero while test error rises). This lab uses a kernel ridge classifier for speed; a kernel SVM shapes the same boundaries with hinge loss and support vectors — see the margin lab for that side of the story. Educational tool.
Where the “support vector” part comes in
This lab drives the kernel with a ridge-style fit for speed; a true SVM combines the same kernel with the maximum-margin objective — find the widest buffer between the classes in the lifted space, supported by only the frontier points. The margin story, hinge loss, and the C parameter get their own hands-on treatment in the advanced track's SVM module. The division of labor is worth remembering: the kernel decides what shapes are expressible; the margindecides which expressible shape to prefer.
Things to try
- • Start at γ = 0.5, λ = 10⁻¹: a clean circle appears — the method invented “radius” on its own.
- • Push γ to 10 and λ to 10⁻⁴: islands form around individual points, training error hits zero, test error climbs. Overfitting, in its most photogenic form.
- • Now γ = 0.2, λ = 10: the boundary oversmooths into a blob that swallows ring points — the underfit corner. The good models live in the middle of both dials, found (as always) by cross-validation.