Module 11 · Trees & ensembles
Neural networks — basis functions that position themselves
Strip the mystique and a one-hidden-layer network is something you already know from Module 5: a linear model on derived features. Each hidden unit computes a simple bent function of the input — classically a sigmoid, today usually a ReLU hinge max(0, ax + b) — and the output is a weighted sum of those bumps. The difference from splines is who chooses the basis: spline knots are fixed by you; a network's hinges have trainable positions and orientations. The model learns its own basis, which is precisely what lets it scale to inputs where hand-placing basis functions is hopeless.
Training is nothing exotic: write down squared (or logistic) loss, differentiate with respect to every weight via the chain rule — that bookkeeping is backpropagation — and descend the gradient. The loss surface is non-convex, so different random starts land in different local solutions; in practice that bothers people less than it sounds like it should, especially with the two standard regularizers: weight decay (the ridge penalty of Module 3 applied to network weights) and early stopping (halt gradient descent before it finishes memorizing).
🎛 Hidden-layer lab
Train MSE
0.076
Test MSE
0.145
Each pale line is one hidden unit's weighted output — a ReLU “hinge” max(0, ax+b) scaled by its output weight. The black curve is just their sum: piecewise-linear, with a bend wherever some unit switches on. A handful of units underfits; dozens can overfit — unless weight decayreins the output weights in. (For speed this lab fixes random hidden weights and solves the output layer exactly; full backprop tunes the hinges' positions too, same picture.) Educational tool.
Reading the lab honestly
For interactivity this lab fixes the hidden weights at random values and solves only the output layer (with decay) in closed form — a “random features” network. Full training would also nudge each hinge's position into the places the data needs them, getting more from fewer units; the qualitative picture — sum of hinges, complexity vs decay — is the same. With enough units, a single hidden layer can approximate any reasonable function (universal approximation); what stacking more layers buys is not raw expressiveness but composition— features built out of features — which is the step from this chapter's networks to modern deep learning.
Things to try
- • Turn on the unit-contribution display at 3 units: you can see each hinge and where it bends. The black output is literally their sum.
- • Go to 80 units with decay at 10⁻⁶: overfitting. Now raise decay and watch the same 80-unit network smooth out — capacity plus regularization beats hand-tuned capacity.
- • Note the fit is piecewise-linear (ReLU) — kinks, not curves. Squint and it approximates the smooth truth anyway; more units make the kinks finer.