Module 4 · Linear models
Two roads to the same straight line
To classify with a linear rule you need a line (in general, a hyperplane) separating the classes. There are two philosophically different ways to get one:
- Generative — LDA. Model each class as a Gaussian cloud with its own mean but a shared covariance, then ask where the two clouds are equally likely. That locus is a straight line with direction Σ⁻¹(μ₁ − μ₀). (Give each class its own covariance and the boundary bends into a quadratic — QDA.)
- Discriminative — logistic regression. Skip the clouds entirely and model the thing you actually need: P(class 1 | x) = 1/(1 + e^−(β₀+βᵀx)), fit by maximum likelihood. It commits to nothing about how x is distributed.
When the Gaussian story is true, LDA uses more information and is a bit more efficient. When it's false, LDA's estimates of means and covariances get polluted by things that shouldn't matter — the lab's third slider adds far-away, confidently-classified points and lets you watch which boundary flinches.
🎛 LDA vs logistic lab
LDA training error
11.4%
models the class densities
Logistic training error
11.4%
models the boundary only
Both methods draw a straight line, but they reason differently: LDA fits a Gaussian to each class and derives the boundary (Σ⁻¹(μ₁−μ₀)); logistic regressionfits the boundary directly by maximum likelihood and assumes nothing about the class shapes. Add far-away class-1 points: they drag LDA's mean and covariance — tilting its line — while logistic barely moves, because points classified confidently get almost no weight in the likelihood. Educational tool.
Why the outliers move one line and not the other
LDA's ingredients are class means and a pooled covariance — everypoint moves them, including points miles from the boundary. Logistic regression's likelihood weights each point by roughly p(1−p), its uncertainty — a point classified with 99.9% confidence contributes almost nothing. So safe, distant points drag LDA's boundary while logistic ignores them. The flip side: when classes are perfectly separable, logistic's coefficients run off to infinity chasing certainty (regularization fixes this), while LDA stays perfectly composed.
Things to try
- • With outliers at 0, the two boundaries nearly coincide — on well-behaved data the philosophical difference is invisible.
- • Add 30 far-away points: the green LDA line tilts toward them; the red logistic line barely moves.
- • Push within-class correlation to ±0.9 and watch both boundaries tilt with the clouds' shared shape — that tilt is the Σ⁻¹ at work.