\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML02.9 · Discriminant analysis

A free, illustrated module of 4 lessons and one interactive in ML02 Supervised Learning: Trees, Ensembles and Kernels: Classes as Gaussian clouds; Linear discriminant analysis; Quadratic discriminant analysis and shrinkage; Fisher’s discriminant and reducing dimension.

ML02.9 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of ML02 Supervised Learning: Trees, Ensembles and Kernels in the Machine Learning track.

Describe each class as a Gaussian cloud with a mean and a covariance matrix, and let Bayes’ rule decide. These four lessons read a cloud’s tilted ellipses off its covariance, show that one shared covariance makes the boundary a straight line (linear discriminant analysis) while one covariance per class makes it a conic (quadratic discriminant analysis), tame the extra parameters with shrinkage, and end with Fisher’s question of which direction best separates the classes, whose answer is the same line and a way to draw many classes in two dimensions.

What you need first. Gaussian naive Bayes and the generative recipe (ML02.6), covariance (M06.10), eigenvectors of a symmetric matrix (M03.3), maximum likelihood (S02.3) and logistic regression (ML01.6). Principal components (M03.6) help in the last lesson.

Lessons

1. Classes as Gaussian clouds ML02.9.1 25 min

   Describe each class as a cloud with a centre and a covariance matrix. Read the cloud’s tilted ellipses off the eigenvectors, measure distance in the cloud’s own units with the Mahalanobis distance, and fit each class of the twelve points by maximum likelihood.

2. Linear discriminant analysis ML02.9.2 25 min

   Give every class the same covariance and the quadratic terms of the log posteriors cancel, leaving a linear boundary. Derive the discriminant line by line, fit it to the twelve points by hand and in NumPy, and compare it with logistic regression.

3. Quadratic discriminant analysis and shrinkage ML02.9.3 25 min

   Let every class keep its own covariance and the boundary bends into an ellipse, a parabola or a hyperbola. Derive quadratic discriminant analysis, fit it to iris by hand and in NumPy, count what it costs in parameters, and use shrinkage when a class has too few points.

4. Fisher’s discriminant and reducing dimension ML02.9.4 25 min

   Ask which direction best separates the classes, with no Gaussians at all: maximise the spread between the projected class means over the spread within. Derive Fisher’s direction, find that it is LDA’s, and use the K - 1 discriminant directions to draw wine in two dimensions beside PCA.

The interactive

ML02.9-X1 · Ellipse explorer. Two or three classes drawn as Gaussian ellipses over sample points. Drag the means, stretch and turn each ellipse, switch between one covariance per class and a shared one, and watch the QDA curve or the LDA line redraw, with Fisher’s direction and the projected histograms beneath. It lives in Quadratic discriminant analysis and shrinkage.