Module 14 · Neighbors & structure
Unsupervised learning — structure without a teacher
Take away the labels and the question changes from “predict y” to “what is this data made of?” Are there natural groups (clustering)? A few directions that carry most of the variation (dimension reduction)? Which variables move together (dependence — Module 17)? These questions are harder to grade than supervised ones — there's no held-out answer to check against — which is why unsupervised results always deserve extra skepticism.
k-meansis the workhorse clusterer, and it's two alternating steps of almost embarrassing simplicity: assign every point to its nearest centroid, then move each centroid to the mean of its members. Each step can only lower the within-cluster sum of squares, so it converges — quickly, and to a local optimum that depends on the start (real implementations restart several times and keep the best). The lab lets you run it one step at a time.
PCAanswers the other question: which single direction (then which second, orthogonal one, and so on) captures the most variance? Algebraically it's the eigenvectors of the covariance matrix; practically it's how a 200-feature dataset becomes 5 interpretable factors — in markets, how a whole yield curve compresses into level, slope and curvature.
🎛 k-means + PCA lab
Inertia (within-cluster SSE)
1076
each iteration can only lower it
At 0 iterations the centroids (black diamonds) sit on arbitrary points and the coloring is nonsense. Step the slider: assign each point to its nearest centroid, then moveeach centroid to its members' mean — watch the diamonds march into the blobs and inertia fall until nothing changes (usually <6 rounds). Try K=2 or K=6: the algorithm always “succeeds” — it cannot tell you K was wrong, which is why you inspect inertia-vs-K elbows and silhouette scores. The grey dashed line is PCA's answer to a different question: not “which groups?” but “which single direction carries the most variance?”. Educational tool.
The honesty problem: choosing K, trusting clusters
k-means never complains: ask for 6 clusters in 3-blob data and you get 6 confident clusters. Standard sanity checks: plot inertia against K and look for the elbow where extra clusters stop paying; compute silhouettescores (is each point closer to its own cluster than the next-best one?); and re-run on bootstrap resamples (Module 8) to see whether the same groups keep re-forming. Hierarchical clustering sidesteps K by building the full merge tree and letting you cut it at any height; Gaussian mixtures replace hard assignments with probabilities. All of them still require you, the analyst, to decide whether the structure found is real.
Things to try
- • Set iterations to 0, then step 1, 2, 3… — most of the convergence happens in the first two rounds; the diamonds sprint, then settle.
- • Set K=6 on this 3-blob data: k-means dutifully splits real clusters in half, and inertia still looks respectable. The algorithm can't tell you K is wrong.
- • Compare the PCA line with the clustering: PCA points along the cloud's longest axis, cheerfully indifferent to the group structure — different question, different answer.