\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

DL01.8 · Anatomy of a training loop

A free, illustrated module of 4 lessons and one interactive in DL01 Neural Network Foundations: The five lines; Epochs, validation and the two modes; First checks when training fails; Checkpoints and a reusable fit function.

DL01.8 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of DL01 Neural Network Foundations in the Deep Learning track.

Every PyTorch training script turns on the same five lines: forward, loss, zero_grad, backward, step. These four lessons take the loop apart: what each line reads and writes and what goes wrong, with numbers, when one is missing; the outer loop of epochs, validation and the two modes; three quick checks that catch most broken loops; and checkpoints that keep the best weights and resume a run exactly, packaged in a fit function used by the rest of the course.

What you need first. Mini-batch gradient descent (ML01.5), backpropagation through a network (DL01.5) and the PyTorch basics of DL01.7: tensors, autograd, nn.Module, the losses and torch.optim.SGD. Learning curves and overfitting (ML01.8) help in the third lesson.

Lessons

1. The five lines DL01.8.1 25 min

   One training iteration in PyTorch is five lines: forward, loss, zero_grad, backward, step. See what each line reads and writes, why their order matters, and what goes wrong, with numbers, when a line is missing or out of place.

2. Epochs, validation and the two modes DL01.8.2 25 min

   Wrap the five lines in an outer loop: an epoch over the training data, a validation pass under torch.no_grad() and model.eval(), epoch metrics weighted by batch size, and training curves drawn from a history dictionary.

3. First checks when training fails DL01.8.3 25 min

   Three quick checks catch most broken training loops before any tuning: a starting loss near ln K, one small batch learned by heart, and a learning rate whose loss falls. Then read the curves: no progress, divergence and a widening gap.

4. Checkpoints and a reusable fit function DL01.8.4 25 min

   Save a run’s whole state with torch.save, keep the weights with the best validation loss, resume a stopped run so that it continues exactly as if it had never stopped, and package it all in a fit function that the next two modules use unchanged.

The interactive

DL01.8-X1 · Loop stepper. Run the five lines of one training iteration one at a time on thirty points and watch the network’s output, the loss, the gradient buffers and the weights change. Skip zero_grad and the gradients pile up until training goes wrong. It lives in The five lines.