\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML03.5 · PCA in depth

A free, illustrated module of 4 lessons and one interactive in ML03 Unsupervised Learning: PCA as compression and as a preprocessing step; Whitening and PCA from distances; Probabilistic PCA and factor analysis; Kernel PCA.

ML03.5 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of ML03 Unsupervised Learning in the Machine Learning track.

Principal component analysis, met in M03.6 as the eigenvectors of a covariance matrix, is taken further here. These four lessons read PCA as an encoder and a decoder whose error is the sum of the dropped eigenvalues, and use it to compress digits and to feed a classifier; whiten data and recover PCA from a table of distances alone; give PCA a probability model whose likelihood can choose the number of components, beside factor analysis; and bend it with a kernel to separate rings that no straight line can.

What you need first. PCA from the covariance matrix and by SVD (M03.6), cross-validation and pipelines (ML01.9), and for the last two lessons the multivariate Gaussian as met in ML03.4 and kernels as met in ML02.8. NumPy and scikit-learn are used in every code section.

Lessons

1. PCA as compression and as a preprocessing step ML03.5.1 25 min

   Read PCA as an encoder that squeezes each example into a few numbers and a decoder that rebuilds it. Derive that the mean squared error of the rebuild is the sum of the dropped eigenvalues, compress the digits, and choose the number of components by cross-validation.

2. Whitening and PCA from distances ML03.5.2 25 min

   Rotate a cloud onto its principal axes and divide by the spreads, and its covariance becomes the identity. Compare PCA and ZCA whitening, find the scores from a small n by n matrix when features outnumber examples, and recover PCA from a table of distances alone by classical MDS.

3. Probabilistic PCA and factor analysis ML03.5.3 25 min

   Give PCA a story: a few hidden numbers, a linear map and round noise. Its maximum-likelihood fit has a closed form built from the eigenvectors, its likelihood chooses the number of components on held-out data, and factor analysis lets each feature keep its own noise.

4. Kernel PCA ML03.5.4 25 min

   Run PCA in the lifted space of a kernel without ever building it. Derive the centred Gram matrix, project training and new points with kernel sums, separate two rings that no line can, and see what the kernel width does, what it costs and why there is no way back.

The interactive

ML03.5-X1 · Variance and error. Turn a line through the mean of a 40-point cloud and watch the variance along it and the mean squared error trade places while their sum stays fixed. Snap to the first principal direction, try a round cloud and one with an outlier, and standardise or whiten the cloud first. It lives in PCA as compression and as a preprocessing step.