\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

DL01.7 · PyTorch fundamentals

A free, illustrated module of 4 lessons and one interactive in DL01 Neural Network Foundations: Tensors; Autograd; nn.Module and layers; Losses and optimizers as objects.

DL01.7 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of DL01 Neural Network Foundations in the Deep Learning track.

Everything built by hand so far, the arrays, the forward pass, the graph that runs backwards, the layers, the losses and the update, has a ready-made counterpart in PyTorch. These four lessons meet each one and check it against the course’s own numbers: a tensor as memory read through a shape and strides, autograd reproducing the hand gradients of the running 2-2-2 network, nn.Module holding parameters with the (out, in) shape the course uses, and losses and optimizers as objects whose defaults and inputs are worth knowing exactly.

What you need first. The forward pass of a multilayer perceptron (DL01.2), the cross-entropy loss (DL01.4), the backward pass by hand on the 2-2-2 network (DL01.5) and the autodiff engine with its Module, Linear and SGD classes (DL01.6). Views and strides in NumPy (P02.3.4) help in the first lesson.

Lessons

1. Tensors DL01.7.1 25 min

   A PyTorch tensor is a NumPy-like array with a dtype, a shape and a device, stored as one row of memory read through strides. Meet the float32 default, memory shared with NumPy, views against copies, broadcasting, and the forward pass of DL01.2 rewritten in torch.

2. Autograd DL01.7.2 25 min

   PyTorch records a graph as each operation runs, exactly as the engine of DL01.6 did, and backward() fills the .grad of every leaf. Reproduce the hand gradients of the running 2-2-2 network, watch gradients accumulate, cut the graph with detach and no_grad, and check a fused loss with gradcheck.

3. nn.Module and layers DL01.7.3 25 min

   nn.Linear stores its weight with shape (out, in) and computes x W-transpose + b, exactly as the course does. Build networks with nn.Sequential and your own Module, count and name their parameters, read a state_dict, and copy NumPy weights in to get the same outputs.

4. Losses and optimizers as objects DL01.7.4 25 min

   nn.MSELoss, nn.BCEWithLogitsLoss and nn.CrossEntropyLoss checked against the losses of DL01.4, why a softmax before CrossEntropyLoss is a silent bug, and torch.optim.SGD as an object that holds the parameters, steps them in place and clears their gradients.

The interactive

DL01.7-X1 · Shapes and strides. Pick the shape of a twelve-number tensor and an operation (transpose, permute, unsqueeze, a slice, a broadcasting add, then optionally view(-1) or contiguous()). The cells are drawn in memory order beside the logical grid, with the shape, the strides and the offset printed, and the read-out says whether the result is a view of the same memory or needs a copy. It lives in Tensors.