\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

M05.8 · Computational graphs and automatic differentiation

A free, illustrated module of 4 lessons and one interactive in M05 Multivariable and Matrix Calculus: Programs as computational graphs; Forward mode: tangents and dual numbers; Reverse mode: adjoints and the tape; Forward or reverse: cost and memory.

M05.8 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of M05 Multivariable and Matrix Calculus in the Mathematics track.

How does a library hand back the slope of a loss in every one of a billion weights for the price of a couple of runs of the model? These four lessons open the box. A program is a graph of elementary steps; forward mode carries a tangent beside every value, with dual numbers doing the bookkeeping; reverse mode records a tape and sweeps adjoints back from the output, giving the whole gradient at once; and the two modes turn out to be the two orders of multiplying a chain of Jacobians, which decides when each one wins.

What you need first. The chain rule in one variable (M04.3), partial derivatives and the gradient with central differences (M05.2), and Jacobians with the multivariable chain rule as sums over paths and as products of Jacobians (M05.4).

Lessons

1. Programs as computational graphs M05.8.1 25 min

   Any program built from elementary operations is a directed acyclic graph, and running it leaves an evaluation trace of intermediate values. Build the trace and the graph of ln x1 + x1 x2 - sin x2 by hand, and see why neither finite differences nor symbolic algebra suits training.

2. Forward mode: tangents and dual numbers M05.8.2 25 min

   Carry beside every value of a program its tangent, the rate at which it moves when the inputs move in a chosen direction. One forward sweep gives a directional derivative, a Jacobian-vector product, and dual numbers with epsilon squared equal to zero make the rules automatic.

3. Reverse mode: adjoints and the tape M05.8.3 25 min

   Run the program forward and record a tape, then sweep backward with adjoints, the derivative of the output with respect to each intermediate value. One backward sweep gives every partial derivative at once; adjoints add where a value is used twice.

4. Forward or reverse: cost and memory M05.8.4 25 min

   Forward mode builds a Jacobian a column per sweep and reverse mode a row per sweep: the two orders of multiplying a chain of Jacobians. Count the work on a chain of three Jacobians, see why a gradient costs a few runs of the model, what reverse mode pays in memory, and how Hessian-vector products combine the two modes.

The interactive

M05.8-X1 · Graph stepper. Step through the graph of ln x1 + x1 x2 - sin x2 node by node. Forward mode fills in values and tangents for a chosen seed; reverse mode fills in values, then adjoints from the output. The read-out counts the sweeps each mode needs for the gradient against the runs of finite differences. It lives in Reverse mode: adjoints and the tape.