\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML01.8 · Overfitting, bias and variance

A free, illustrated module of 4 lessons and one interactive in ML01 Machine Learning Foundations: Too simple and too flexible; Bias and variance; Learning curves; What to do about it.

ML01.8 L1 · Core 4 lessons · about 105 min Live · in review

Part of ML01 Machine Learning Foundations in the Machine Learning track.

A model can fail in two opposite ways: it can be too simple to follow the pattern, or flexible enough to follow the noise. These four lessons fit polynomials of rising degree to twelve noisy points and watch both failures happen, split the expected error into bias squared, variance and noise and derive the split step by step, read the two failures off a learning curve, and match each remedy (more data, a simpler or richer model, better features, a penalty) to the failure it cures.

What you need first. Training, validation and test sets and the generalisation gap (ML01.2), and linear regression with several features solved by least squares (ML01.4). The mean and the variance of a list of numbers. A little NumPy helps for the code.

Lessons

1. Too simple and too flexible ML01.8.1 25 min

   Fit polynomials of rising degree to twelve noisy points. A straight line misses the pattern, a degree-11 curve passes through every point and learns the noise, and only held-back examples can say where between them to stop.

2. Bias and variance ML01.8.2 30 min

   Draw the training set again and the fitted curve changes. The expected squared error of a prediction splits into three parts: the squared bias of the average fit, the variance of the fits around it, and the noise. Derive the split step by step and measure each part.

3. Learning curves ML01.8.3 25 min

   Plot the training error and the validation error against the number of training examples. Where the two curves level off shows the bias, and the gap between them shows the variance. Derive the curve of the simplest model and learn to read the rest.

4. What to do about it ML01.8.4 25 min

   More data, a simpler or a richer model, better features, a penalty: each remedy shrinks one part of the error and leaves the others alone. Diagnose a model from its training and validation errors, pick the remedy that fits, and watch each one work or fail on one example.

The interactive

ML01.8-X1 · Polynomial fitter. Choose the degree of a polynomial and the number of training points, then draw new noisy training sets from the same source. The fitted curves pile up: where they miss together you see bias, where they disagree you see variance, and a chart shows training and test error against the degree. It lives in Bias and variance.