Skip to main content

ML03.10 · Evaluating clusters and embeddings

A free, illustrated module of 4 lessons and one interactive in ML03 Unsupervised Learning: Internal indices: the silhouette and its relatives; External indices: comparing with known groups; Is there any structure? Stability and the gap statistic; Evaluating embeddings.

ML03.10 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of ML03 Unsupervised Learning in the Machine Learning track.

Every clustering algorithm returns clusters and every embedding returns a picture, whether or not the data hold any structure. These four lessons build the tools for judging them honestly: internal indices (the silhouette, Calinski-Harabasz and Davies-Bouldin) that score tightness and separation, external indices (the adjusted Rand index and adjusted mutual information) that compare with known groups without rewarding chance, the gap statistic and stability for asking whether there is any structure at all, and trustworthiness, continuity and Shepard diagrams for asking what an embedding has kept.

What you need first. k-means and its objective J (ML03.1), DBSCAN (ML03.3) and Gaussian mixtures with BIC (ML03.4) for the clusterings that are judged; PCA, t-SNE and UMAP (ML03.5 and ML03.6) for the last lesson. Counting pairs with “n choose 2” and a first idea of entropy help in the second lesson.

Lessons

  1. Internal indices: the silhouette and its relatives ML03.10.1 25 min

    With no labels, a clustering can only be judged by how tight and how separated its clusters are. Compute the silhouette of every point, read a silhouette plot, define the Calinski-Harabasz and Davies-Bouldin indices, and see why all of them favour round clusters.

  2. External indices: comparing with known groups ML03.10.2 25 min

    When reference labels exist, compare a clustering with them through the contingency table, without caring how clusters are numbered. Purity, the Rand index over pairs, the adjusted Rand index that scores chance as 0, and mutual information with its normalised and adjusted forms.

  3. Is there any structure? Stability and the gap statistic ML03.10.3 25 min

    Every clustering algorithm returns clusters, even from uniform noise. Compare log W with its value on uniform reference data (the gap statistic), check whether the clusters survive resampling (stability), and learn where each test, and BIC, can still be fooled.

  4. Evaluating embeddings ML03.10.4 25 min

    An embedding is judged by what it keeps. Measure trustworthiness and continuity from neighbour ranks, the share of nearest neighbours preserved, the Shepard diagram of distances and a k-NN classifier in the map, for PCA, Isomap, t-SNE and UMAP on 500 digits.

The interactive

ML03.10-X1 · Cluster scorecard. Choose a point set (three blobs, touching blobs, moons or uniform noise) and K from 2 to 6. k-means clusters it, every point shows its silhouette as a bar, and the read-out gives the mean silhouette, the Davies-Bouldin index, the adjusted Rand index against the true groups and a stability score over subsamples. On the noise every index still reports clusters. It lives in Is there any structure? Stability and the gap statistic.

Back to top