\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

DL01.10 · First project: an image classifier

A free, illustrated module of 4 lessons and one interactive in DL01 Neural Network Foundations: Frame the task and set the bar; Train the network; Test once and study the errors; What the network learned, and what it cannot see.

DL01.10 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of DL01 Neural Network Foundations in the Deep Learning track.

The course’s first full project: a network that reads handwritten digits. These four lessons frame the task and set the bar with two baselines, train a 64-32-10 network after checking the loop and choose its width and learning rate on the validation set alone, open the test set once and study the mistakes with honest error bars, and finally open the network up, ending with the experiment that shows what a multilayer perceptron cannot see: shuffle every image’s pixels and it does just as well.

What you need first. The training loop, checks and fit function of DL01.8, the digits split and the loaders of DL01.9, and the losses of DL01.4. Accuracy, recall and the confusion matrix (ML01.7), and the standard error of a test score (ML01.2.3) are used throughout.

Lessons

1. Frame the task and set the bar DL01.10.1 25 min

   Before any network is trained, say what the task is, how it will be scored and what score would count as success. Examine the 1,797 digit images, score with accuracy and per-class recall, and set two baselines on the validation set: the majority class and logistic regression.

2. Train the network DL01.10.2 25 min

   Build the 64-32-10 network, run the two first checks before trusting the loop, train it with the fit function of DL01.8.4, and choose the width and learning rate from a grid of nine runs using the validation set alone, with the one-standard-error rule.

3. Test once and study the errors DL01.10.3 25 min

   Open the test set for the one and only time: report the network’s accuracy with a binomial standard error, compare it with logistic regression image by image, and read the confusion matrix, the most confused pairs and the misclassified images.

4. What the network learned, and what it cannot see DL01.10.4 25 min

   Open the trained network up: its 32 first-layer weight vectors drawn as images and its hidden activations projected with PCA. Then shuffle the pixels of every image the same way, retrain, and see accuracy stay put, which proves the network never used which pixels are neighbours.

The interactive

DL01.10-X1 · Digit sketchpad. Paint a digit on an 8 by 8 pad, or load a test image, and watch the class probabilities and the hidden activations of the network trained in this module. Shuffle the pixels and see a retrained network read the noise as well as the original reads the digit. It lives in What the network learned, and what it cannot see.