\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML02.7 · Maximum margin and support vector machines

A free, illustrated module of 4 lessons and one interactive in ML02 Supervised Learning: Trees, Ensembles and Kernels: The widest street; The hard-margin problem and its dual; Soft margins and the hinge loss; Linear SVMs in practice.

ML02.7 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of ML02 Supervised Learning: Trees, Ensembles and Kernels in the Machine Learning track.

When two classes can be split by a straight line, infinitely many lines do it. The support vector machine chooses the centre of the widest street between the classes. These four lessons derive the width of the street, solve for it with Lagrange multipliers and find that only a few support vectors matter, relax it with slack variables into the hinge loss when no line is perfect, and then use linear SVMs on real data with scaling, a cross-validated C and a regression version that fits a tube.

What you need first. Logistic regression and its decision boundary (ML01.6), dot products and projection onto a line (M02.3), gradient descent (ML01.5), and ridge regularisation with cross-validation (ML01.9). Lagrange multipliers are introduced just as far as needed in lesson 2.

Lessons

1. The widest street ML02.7.1 25 min

   When two classes can be split by a straight line, infinitely many lines do it. The maximum-margin line is the one furthest from the closest points: the centre of the widest street between the classes. Derive the distance to a line and find the street for six points by hand.

2. The hard-margin problem and its dual ML02.7.2 25 min

   The widest street becomes an optimisation problem: minimise half the squared length of w while every point stays off the street. Lagrange multipliers turn it into a dual problem whose solution gives each point a weight, zero for all but the support vectors.

3. Soft margins and the hinge loss ML02.7.3 25 min

   Real classes overlap, and then no street is empty. Slack variables let points into the street or across the line at a price C, the dual gains a ceiling C on every weight, and eliminating the slacks reveals the SVM as ridge-penalised hinge loss, trainable by subgradient descent.

4. Linear SVMs in practice ML02.7.4 25 min

   A linear SVM on real data: why the features must be scaled first and inside a pipeline, how to choose C by cross-validation, how the SVM compares with logistic regression, and how support vector regression fits a tube instead of a line.

The interactive

ML02.7-X1 · Margin explorer. Drag up to twenty points of two classes and watch the widest street solve itself live, with its support vectors ringed and their dual weights listed. Switch to a soft margin and slide C to see the slacks and the points that enter and leave the street. It lives in Soft margins and the hinge loss.