\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML02.10 · Multi-class, multi-label and ordinal problems

A free, illustrated module of 4 lessons and one interactive in ML02 Supervised Learning: Trees, Ensembles and Kernels: Many classes at once: softmax regression; One-vs-rest, one-vs-one and output codes; Multi-label: several labels per example; Ordinal targets.

ML02.10 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of ML02 Supervised Learning: Trees, Ensembles and Kernels in the Machine Learning track.

Real labels are rarely a single yes or no. These four lessons extend classification to many classes, to several labels per example and to classes that come in an order. Softmax regression gives logistic regression a score per class; one-vs-rest, one-vs-one and output codes build many classes out of binary learners such as the SVM, and SAMME does it for AdaBoost; binary relevance and classifier chains predict sets of labels; and the proportional-odds model cuts one score with ordered thresholds. Each lesson ends with the metrics that judge its kind of prediction fairly.

What you need first. Logistic regression, the softmax and the log loss (ML01.6), gradient descent (ML01.5), precision, recall, F1 and the mean absolute error (ML01.7), support vector machines (ML02.7), AdaBoost (ML02.3) and random forests (ML02.2).

Lessons

1. Many classes at once: softmax regression ML02.10.1 25 min

   Logistic regression grows into softmax regression: one score per class, the softmax turns the scores into probabilities, and the cross-entropy gradient is prediction minus one-hot label times the input. Then the K by K confusion matrix and its macro, micro and weighted averages.

2. One-vs-rest, one-vs-one and output codes ML02.10.2 25 min

   Build a K-class classifier out of binary ones. One-vs-rest trains K and takes the largest score, one-vs-one trains K(K-1)/2 and votes, and output codes add spare bits that correct mistakes. Plus SAMME, the multi-class AdaBoost, and its extra ln(K-1).

3. Multi-label: several labels per example ML02.10.3 25 min

   When a photo can be tagged sea, people and dog at once, the target is a binary matrix. Binary relevance trains one classifier per label, a classifier chain lets later labels see earlier ones, and subset accuracy, Hamming loss and micro and macro F1 each judge a different thing.

4. Ordinal targets ML02.10.4 25 min

   Ratings, grades and severity levels come in an order. The proportional-odds model cuts one score with increasing thresholds; its likelihood is derived and maximised from scratch on the diabetes data in four grades, beside the Frank and Hall reduction and metrics that respect the order.

The interactive

ML02.10-X1 · Reduction explorer. Three classes of points with a linear classifier per class, per pair, or all trained together. Switch between one-vs-rest, one-vs-one and softmax and watch the regions redraw, with the places where the binary answers abstain, overlap or tie hatched and measured. It lives in One-vs-rest, one-vs-one and output codes.