Module 12 · Trees & ensembles
Support vector machines — the widest street
Module 4's classifiers ask “which line separates the classes?” — but many lines do. The SVM asks a sharper question: which line leaves the most room? Picture the boundary as the centerline of a street; the SVM finds the orientation that makes the street as wide as possible before touching a data point. Wide margin = predictions that survive a little jitter in the inputs — a geometric argument for generalization.
Real classes overlap, so the modern (“soft margin”) version lets points violate the street for a price: minimize ½||w||² + C·Σ slack. The single dial Csets the exchange rate between street width and violations — it is this method's bias-variance knob.
The structural surprise: the solution depends only on the points that touch or violate the margin — the support vectors. The comfortable majority, classified with room to spare, could be deleted without moving the boundary at all. Equivalently, the SVM minimizes hinge loss, which is exactly zero for comfortably-correct points; compare logistic loss, which never quite stops caring.
🎛 Margin lab
Margin width
0.41
2/||w|| across the street
Support vectors
6 / 90
points on or inside the margin
Training errors
5
on the wrong side entirely
The SVM maximizes the gap between the dashed margin lines while paying C for every point that crosses its line. Points with a red halo are the support vectors— only they touch the solution; delete any other point and the boundary wouldn't move an inch. Big C: narrow, twitchy margin that hates violations. Small C: wide, calm margin that tolerates a few. Kernels reuse this exact machinery after implicitly mapping x into a richer feature space. Educational tool.
The kernel trick, in one paragraph
Everything the SVM computes involves data points only through inner products xᵢ·xⱼ. So replace that inner product with a kernel function K(xᵢ, xⱼ) — Gaussian, polynomial — and you have implicitly mapped the data into a vastly higher-dimensional feature space and found the widest street there, without ever writing the mapping down. A straight boundary in that hidden space bends into curves and islands back in the original one. Same trick, same caveat as always: a richer space means C and the kernel width must be tuned by cross-validation, or the islands will form around individual noise points.
Things to try
- • Sweep log C from −2 to +2: watch the street narrow and the support-vector count fall. At small C, half the dataset is load-bearing; at large C, only the frontier is.
- • Raise the overlap and note the SVM's composure: it doesn't chase a perfect separation that doesn't exist — it prices the violations and moves on.
- • Count halos vs errors: support vectors include correct points inside the margin, not just mistakes. “Close calls” are what define the boundary.