\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML03.8 · Density estimation

A free, illustrated module of 4 lessons and one interactive in ML03 Unsupervised Learning: From histograms to kernel density estimates; Choosing the bandwidth; Densities in several dimensions; Mean shift: clustering by climbing the density.

ML03.8 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of ML03 Unsupervised Learning in the Machine Learning track.

A density estimate turns a sample into a smooth curve of area 1 that says where the data crowd. These four lessons build the kernel density estimate as a bump on every point, derive the bias and variance that the bandwidth trades, let leave-one-out likelihood choose it, carry the estimate into several dimensions, where the data needed grow explosively, and climb its hills with mean shift to find clusters, with every number checked from scratch and against SciPy and scikit-learn.

What you need first. Histograms on the density scale (S01.4), densities and the normal curve (M06.6), Taylor polynomials (M04.8), bias, variance and mean squared error of an estimator (S02.2), and Gaussian mixtures with the six values of ML03.4. Bayes’ rule (M06.4) and DBSCAN (ML03.3) help in the last two lessons.

Lessons

1. From histograms to kernel density estimates ML03.8.1 25 min

   Turn a sample into a smooth curve of area 1 by putting a small bump on every point and adding the bumps. See why histograms depend on where the bins start, prove that a kernel estimate has area 1, and compute one by hand, from scratch and with SciPy and scikit-learn.

2. Choosing the bandwidth ML03.8.2 25 min

   A small bandwidth follows the noise and a large one smooths away the shape. Derive the bias and variance behind that trade-off, the best bandwidth shrinking like n to the minus one fifth, the rules of thumb of Silverman, and leave-one-out likelihood as a choice made by the data.

3. Densities in several dimensions ML03.8.3 25 min

   Carry the kernel estimate into the plane with product kernels and standardised features, draw it as contours over two iris features, see how fast the data needed grow with the dimension, sample new points from an estimate, and build a classifier from one estimate per class.

4. Mean shift: clustering by climbing the density ML03.8.4 25 min

   Take the hills of a kernel density estimate as clusters. Derive the gradient of a Gaussian estimate, show that moving a point to the kernel-weighted mean of the data is a gradient-ascent step with an adaptive size, and cluster the six values and seeded blobs from scratch and with MeanShift.

The interactive

ML03.8-X1 · Bandwidth explorer. Drag up to twenty points on a line, each with its own kernel bump, and watch their sum. Switch between Gaussian, Epanechnikov and tophat kernels, slide the bandwidth or jump to Silverman’s rule and the leave-one-out best, and compare with a histogram of the same width. It lives in Choosing the bandwidth.