\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

M05.7 · Matrix calculus identities

A free, illustrated module of 4 lessons and one interactive in M05 Multivariable and Matrix Calculus: Shapes and the first identities; Differentials: the reliable method; Gradients of a dense layer and of softmax with cross-entropy; Determinant, inverse and the log-determinant.

M05.7 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of M05 Multivariable and Matrix Calculus in the Mathematics track.

Training a model means differentiating a loss with respect to vectors and matrices of weights. These four lessons fix the shape of every such derivative (gradient, Jacobian, gradient with respect to a matrix), give one reliable method for finding it (write the differential, rearrange it into a trace, read off the gradient), and use that method on least squares and ridge, on the dense layer and softmax with cross-entropy that every network is built from, and on the determinant, the inverse and the log-determinant behind Gaussian fitting and normalising flows.

What you need first. Partial derivatives and gradients in many variables (M05.2), Jacobians and the chain rule (M05.4), index notation and the trace identities of M03.10.2, determinants and inverses (M02.7 and M02.8), and least squares through the normal equations (M03.7.1).

Lessons

1. Shapes and the first identities M05.7.1 25 min

   A derivative in matrix calculus is a block of numbers with a definite shape. Learn the three kinds (gradient, Jacobian, gradient by a matrix), the two layout conventions, and derive the first identities in index notation.

2. Differentials: the reliable method M05.7.2 25 min

   Write the first-order change df in terms of the change dX, rearrange it into tr(G-transpose dX), and read off the gradient G. Derive least squares, ridge and three matrix gradients this way, without a single index.

3. Gradients of a dense layer and of softmax with cross-entropy M05.7.3 25 min

   Derive with differentials the three gradients of a dense layer, X-transpose G, G W-transpose and the column sums of G, then the softmax Jacobian and the famous p minus y, and check every shape on a batch of two examples.

4. Determinant, inverse and the log-determinant M05.7.4 25 min

   Derive with differentials the change of an inverse, -X^(-1) dX X^(-1), and of a determinant, det X tr(X^(-1) dX), so that the gradient of log det X is X^(-T). Then use them to find the maximum-likelihood covariance of a Gaussian.

The interactive

M05.7-X1 · Gradient heatmap. Pick a function of a small matrix X, such as tr(AX), a-transpose X b or ln det X, and move the entries of X with sliders. The slope measured by nudging each entry sits beside the formula’s gradient, cell by cell; switch to a common mistake and the read-out flags a wrong shape or the cells that disagree. It lives in Differentials: the reliable method.