Skip to main content

Losses and optimizers as objects

nn.MSELoss, nn.BCEWithLogitsLoss and nn.CrossEntropyLoss checked against the losses of DL01.4, why a softmax before CrossEntropyLoss is a silent bug, and torch.optim.SGD as an object that holds the parameters, steps them in place and clears their gradients.
DL01.7.4L2 Intermediate25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: DL01.7.3 nn.Module and layers DL01.4 Loss functions and maximum likelihood

What you will be able to do
  • Compute nn.MSELoss, nn.BCEWithLogitsLoss and nn.CrossEntropyLoss from their formulas, including the default reduction=‘mean’.
  • State what nn.CrossEntropyLoss expects (logits of shape (N, C) and int64 labels of shape (N,)) and show why a softmax before it is a bug.
  • Explain what torch.optim.SGD holds and what step() and zero_grad() change.
  • Match one SGD step in PyTorch to the same step computed by hand, parameter by parameter.

The question

Hand nn.CrossEntropyLoss the probabilities of your network instead of its logits, and nothing complains: the loss comes out as a sensible-looking number and training even makes progress. Then the loss stops falling at 0.313 and never goes lower. What does the loss object do inside, and what does the optimizer object do to your parameters when you call step()?

From the last lesson

1. How many parameters does the 2-16-16-2 network have, and how is the count made?

Show answer

354: each layer holds \({n_l (n_{l-1} + 1)}\) numbers, so \({16 \cdot 3} + {16 \cdot 17} + {2 \cdot 17} = 48 + 272 + 34\).

2. What does model.parameters() return for a module that keeps its layers in a plain Python list?

Show answer

Nothing from those layers: a list is not registered, so its layers’ parameters are not found.

Intuition

In DL01.4 a loss was a formula you wrote as a function. In PyTorch it is an object, made once with its settings and then called like a function: loss_fn = nn.CrossEntropyLoss(), then loss_fn(logits, labels). The optimizer is an object too, and the more surprising one. It is made from the network’s parameters, torch.optim.SGD(model.parameters(), lr=0.1), and it keeps references to those very tensors, not copies of their values.

Two parts. On the left, the loss object: vermillion logits z of shape (N, C) and black integer labels of shape (N,) enter a box nn.CrossEntropyLoss, inside which three small steps are drawn, log-softmax, pick minus log p of the true class, and mean over the batch, and a reddish-purple scalar L comes out. On the right, the optimizer object: a box torch.optim.SGD with lr 0.1 has four thin arrows, labelled references, to the orange parameter tensors W1, b1, W2 and b2 of the network, each with a green dot-grad slot. Below it, two notes: step() does p minus equals lr times p.grad on each, in place; zero_grad() sets each p.grad to None.
DL01.7.4-F1

The two objects of a training step. The loss object turns logits and labels into one number, doing the softmax itself. The optimizer holds references to the network’s own parameter tensors: step() changes them in place, and zero_grad() clears their gradients.

Because the optimizer holds references, nothing is ever handed back. After optimizer.step(), the model already has its new weights, since the model and the optimizer point at the same tensors. That is also why the optimizer must be made from the parameters of the model you will train, after that model exists.

Build it up

Step 1. Mean squared error. nn.MSELoss() returns the mean of \({(\hat{y}_i - y_i)^2}\), with the \(1/n\) and no one half, as in ML01. On the four points \(x = 1, 2, 3, 4\) with targets \(2, 3, 5, 6\), the least-squares line \(1.4x + 0.5\) predicts \(1.9, 3.3, 4.7, 6.1\); the squared residuals are \(0.01, 0.09, 0.09, 0.01\), and the loss is \(0.2 / 4 = 0.05\).

Step 2. Binary cross-entropy from a logit. nn.BCEWithLogitsLoss() takes the logit \(z\) (the pre-activation, before any sigmoid) and a target \(y\) of 0 or 1 as a float. It computes the log loss of DL01.4.2 in its stable form, \(\operatorname{softplus}(-z)\) when \(y = 1\) and \(\operatorname{softplus}(z)\) when \(y = 0\), where \({\operatorname{softplus}(u)} = {\ln(1 + e^u)}\).

Two panels. Left, nn.MSELoss for one example: a reddish-purple parabola of the loss against the prediction y-hat, with its lowest point, zero, at the target y marked by a black tick. Right, nn.BCEWithLogitsLoss for one example against the logit z from minus 6 to 6: a reddish-purple curve softplus of minus z for y equals 1, falling towards zero on the right and rising like a straight line of slope minus 1 on the left, and a dashed curve softplus of z for y equals 0, its mirror image. Both curves cross at z equals 0, where the loss is ln 2, about 0.693.
DL01.7.4-F2

The two simplest loss objects, for one example. Squared error is smallest at the target. Binary cross-entropy takes the logit: for \(y = 1\) it is \(\operatorname{softplus}(-z)\), near zero for large positive \(z\) and growing like \(|z|\) when the model is confidently wrong.

Step 3. Categorical cross-entropy. nn.CrossEntropyLoss() takes logits of shape \((N, C)\) and integer class labels of shape \((N,)\) with dtype int64. For each example it computes the log-softmax of the logits and keeps minus the entry of the true class, which is \(-z_k + \ln \sum_j e^{z_j}\), the form derived in DL01.4.3. It is the same as nn.LogSoftmax followed by nn.NLLLoss.

Step 4. Reduction. Every loss object takes reduction: "mean" (the default) averages the per-example losses, "sum" adds them, and "none" returns them all. With the mean, the gradient of each example carries a factor \(1/N\), the \(1/n\) of \({(\mat{P} - \mat{Y})/n}\) in DL01.5.2.

Step 5. The optimizer. torch.optim.SGD(params, lr=0.1) keeps the parameters in a list (in optimizer.param_groups). step() sets \(\theta \leftarrow \theta - \eta\,\theta.\mathrm{grad}\) for each, in place and outside the graph. zero_grad() clears every .grad; by default it sets it to None rather than filling it with zeros, which saves memory. A parameter whose .grad is None is simply skipped by step(), so a forgotten backward() gives no step at all, not a wrong one.

The formal statement

For a batch of \(N\) examples, each loss object first computes one loss \(\ell_i\) per example,

\[ \begin{aligned} \ell^{\text{MSE}}_i &= (\hat{y}_i - y_i)^2, \\ \ell^{\text{BCE}}_i &= \operatorname{softplus}\big((1 - 2y_i)\, z_i\big), \\ \ell^{\text{CE}}_i &= -z_{i, k_i} + \ln \textstyle\sum_{j=1}^{C} e^{z_{ij}}, \end{aligned} \]

and with the default reduction="mean" returns their mean,

\[ L = \frac{1}{N} \sum_{i=1}^{N} \ell_i. \]

The BCE line packs both cases into one: \(1 - 2y\) is \(-1\) when \(y = 1\) and \(+1\) when \(y = 0\). In the CE line \(k_i\) is the label of example \(i\), and \(z_{ij}\) its logit for class \(j\).

Four reddish-purple bars for the per-example squared errors of the line 1.4x plus 0.5 on the four points: 0.01, 0.09, 0.09 and 0.01, labelled reduction none returns these four. To the right, a stacked bar of height 0.2 labelled reduction sum, and a short bar of height 0.05 labelled reduction mean, the default. A note says the mean divides by N, so each example's gradient carries a factor 1 over N.
DL01.7.4-F3

The reduction argument on the four points of ML01. "none" keeps the four squared errors, "sum" adds them to 0.2, and the default "mean" divides by 4 to give 0.05.

One step of plain SGD with learning rate \(\eta\) updates every parameter tensor \(\theta\) the optimizer holds by

\[ \theta \;\leftarrow\; \theta - \eta\, \theta.\mathrm{grad}. \]

This is the update of ML01.5 and of your DL01.6.3 optimizer. With its defaults (momentum=0, dampening=0, weight_decay=0, nesterov=False) PyTorch’s SGD does exactly this and nothing more. Its default learning rate is \(10^{-3}\); this course always passes lr explicitly.

Why a softmax before CrossEntropyLoss is a bug. The loss applies its own log-softmax. Give it probabilities \(\vect{q} = \operatorname{softmax}(\vect{z})\) instead of logits, and it computes \(-q_k + \ln \sum_j e^{q_j}\), the cross-entropy of a second softmax. Every \(q_j\) lies in \([0, 1]\), so this can never reach 0. With \(C = 2\) classes, write \(t = q_k\), so the other entry is \(1 - t\):

\[ \begin{aligned} f(t) &= -t + \ln\big(e^{t} + e^{1 - t}\big), \\ f'(t) &= -1 + \frac{e^{t} - e^{1 - t}}{e^{t} + e^{1 - t}} \\ &= \frac{-2\,e^{1 - t}}{e^{t} + e^{1 - t}} < 0. \end{aligned} \]

So \(f\) falls as the model grows more confident in the right class, but only as far as \(f(1) = -1 + {\ln(e + 1)} \approx 0.3133\), however good the network is. For the ten classes of DL01.10 the floor is \(-1 + {\ln(e + 9)} \approx 1.461\).

Worked example

One SGD step on the running network. DL01.7.2 found the gradients of the 2-2-2 network of DL01.5.1 at \(\vect{x} = (1, 2)\) with true class 2: the loss is \(1.3133\) and

\[ \begin{aligned} \pd{L}{\mat{W}^{(2)}} &= \begin{bmatrix} 0.3655 & 0.7311 \\ -0.3655 & -0.7311 \end{bmatrix}, \\ \pd{L}{\vect{b}^{(2)}} &= (0.7311, -0.7311), \\ \pd{L}{\mat{W}^{(1)}} &= \begin{bmatrix} 1.4621 & 2.9242 \\ 0.3655 & 0.7311 \end{bmatrix}, \\ \pd{L}{\vect{b}^{(1)}} &= (1.4621, 0.3655). \end{aligned} \]

With \(\eta = 0.1\) each parameter moves by minus a tenth of its gradient:

\[ \begin{aligned} \mat{W}^{(2)} &\leftarrow \begin{bmatrix} 0.9634 & 0.9269 \\ -0.9634 & 0.5731 \end{bmatrix}, \\ \vect{b}^{(2)} &\leftarrow (-0.5731,\; 0.0731), \\ \mat{W}^{(1)} &\leftarrow \begin{bmatrix} 0.8538 & -0.7924 \\ -0.5366 & 0.9269 \end{bmatrix}, \\ \vect{b}^{(1)} &\leftarrow (0.3538,\; -0.5366). \end{aligned} \]

For example the top-left weight of \(\mat{W}^{(2)}\) moves from 1 to \({1 - 0.1 \cdot 0.3655}\), and \(b^{(1)}_1\) from \(0.5\) to \({0.5 - 0.1 \cdot 1.4621}\). Run the network again. The first hidden unit now has \(z^{(1)}_1 \approx 0.8538 - 1.5848 + 0.3538 \approx -0.377\): the step has switched it off. The second has \(z^{(1)}_2 \approx -0.5366 + 1.8538 - 0.5366 \approx 0.781\), and the logits are about \({(0.1505, 0.5205)}\). The true class now leads by \(0.3700\), and the loss fell from \(1.3133\) to \({\ln(1 + e^{-0.3700})} \approx 0.5252\).

A trace of one SGD step with learning rate 0.1 on the running 2-2-2 network, one row per parameter and three columns: before in orange, plus minus 0.1 times the gradient in green, equals after in orange. W2: rows [1, 1] and [-1, 0.5], plus rows [-0.037, -0.073] and [0.037, 0.073], equals rows [0.963, 0.927] and [-0.963, 0.573]. b2: [-0.5, 0] plus [-0.073, 0.073] equals [-0.573, 0.073]. W1: rows [1, -0.5] and [-0.5, 1], plus rows [-0.146, -0.292] and [-0.037, -0.073], equals rows [0.854, -0.792] and [-0.537, 0.927]. b1: [0.5, -0.5] plus [-0.146, -0.037] equals [0.354, -0.537]. A note says that after the step z1 equals (minus 0.377, 0.781), so hidden unit 1 is now off, and the reddish-purple line at the bottom reads loss 1.313 falls to 0.525.
DL01.7.4-F4

One step, parameter by parameter: before, minus \(\eta\) times the gradient, after. The loss drops from 1.313 to 0.525, and the step also pushes the first hidden unit below zero, so it is off for this input afterwards.

Code

The three loss objects, each against its formula from DL01.4 written from scratch.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from sklearn.datasets import make_moons

torch.manual_seed(0)
torch.set_num_threads(1)
dtype = torch.float64

x4 = torch.tensor([1.0, 2.0, 3.0, 4.0], dtype=dtype)
y4 = torch.tensor([2.0, 3.0, 5.0, 6.0], dtype=dtype)
pred = 1.4 * x4 + 0.5
mse = nn.MSELoss()(pred, y4)
per_example = nn.MSELoss(reduction="none")(pred, y4)
print(f"MSE {mse.item():.4f}", np.round(per_example.numpy(), 4).tolist(),
      f"sum {nn.MSELoss(reduction='sum')(pred, y4).item():.4f}")

softplus = lambda u: torch.log1p(torch.exp(u))           # fine for these small logits
z_b = torch.tensor([2.0, -1.0], dtype=dtype)
y_b = torch.tensor([1.0, 0.0], dtype=dtype)
bce = nn.BCEWithLogitsLoss()(z_b, y_b)
print(f"BCE {bce.item():.4f}  by hand {softplus((1 - 2 * y_b) * z_b).mean().item():.4f}")

Z = torch.tensor([[3.0, 2.0], [0.5, -1.0], [-2.0, 1.0]], dtype=dtype)
k = torch.tensor([1, 0, 1])
ce = nn.CrossEntropyLoss()(Z, k)
by_hand = (-Z[torch.arange(3), k] + torch.logsumexp(Z, dim=1)).mean()
print(f"CE {ce.item():.4f}  by hand {by_hand.item():.4f}  labels {k.dtype}")
MSE 0.0500 [0.01, 0.09, 0.09, 0.01] sum 0.2000
BCE 0.2201  by hand 0.2201
CE 0.5211  by hand 0.5211  labels torch.int64

Now the bug of the hook: a softmax before the loss. On the running logits \((1, 0)\) with true class 2 the right loss is \(1.3133\); the doubled softmax reports \(0.9507\), and its gradient is about a third of the true one.

z = torch.tensor([[1.0, 0.0]], dtype=dtype, requires_grad=True)
label = torch.tensor([1])
right = nn.CrossEntropyLoss()(z, label)
(g_right,) = torch.autograd.grad(right, z)
wrong = nn.CrossEntropyLoss()(F.softmax(z, dim=1), label)
(g_wrong,) = torch.autograd.grad(wrong, z)
print(f"right {right.item():.4f} grad {np.round(g_right.numpy(), 4).tolist()}")
print(f"wrong {wrong.item():.4f} grad {np.round(g_wrong.numpy(), 4).tolist()}")

margins = torch.linspace(-10, 10, 201, dtype=dtype)
logits = torch.stack([torch.zeros_like(margins), margins], dim=1)     # class 2 ahead by the margin
labels = torch.ones(201, dtype=torch.int64)
floor = nn.CrossEntropyLoss(reduction="none")(F.softmax(logits, dim=1), labels).min().item()
print(f"lowest loss with the extra softmax {floor:.4f}; bound {-1 + np.log(np.e + 1):.4f}; ten classes {-1 + np.log(np.e + 9):.3f}")
right 1.3133 grad [[0.7311, -0.7311]]
wrong 0.9507 grad [[0.2412, -0.2412]]
lowest loss with the extra softmax 0.3133; bound 0.3133; ten classes 1.461

The optimizer and its references. One step on the running network, checked against the worked example.

start = {"W1": [[1.0, -0.5], [-0.5, 1.0]], "b1": [0.5, -0.5], "W2": [[1.0, 1.0], [-1.0, 0.5]], "b2": [-0.5, 0.0]}
params = {name: nn.Parameter(torch.tensor(v, dtype=dtype)) for name, v in start.items()}
x = torch.tensor([1.0, 2.0], dtype=dtype)

def running_loss():
    hidden = torch.relu(params["W1"] @ x + params["b1"])
    return nn.CrossEntropyLoss()((params["W2"] @ hidden + params["b2"]).unsqueeze(0), label)

optimizer = torch.optim.SGD(params.values(), lr=0.1)
print(optimizer.param_groups[0]["params"][2] is params["W2"])     # a reference, not a copy

loss_before = running_loss()
optimizer.zero_grad()
loss_before.backward()
grads = {name: p.grad.clone() for name, p in params.items()}
optimizer.step()
loss_after = running_loss()
for name, p in params.items():
    print(name, np.round(p.detach().numpy(), 4).tolist())
print(f"loss {loss_before.item():.4f} -> {loss_after.item():.4f}")
optimizer.zero_grad()
print("after zero_grad:", params["W1"].grad)
True
W1 [[0.8538, -0.7924], [-0.5366, 0.9269]]
b1 [0.3538, -0.5366]
W2 [[0.9634, 0.9269], [-0.9634, 0.5731]]
b2 [-0.5731, 0.0731]
loss 1.3133 -> 0.5252
after zero_grad: None

Finally the 2-16-16-2 network and the very first step of the training run of DL01.6.4: the same 150 training points, the same starting weights from np.random.default_rng(0), the same first batch of 32 and lr=0.2. PyTorch takes the step, and the same step is done with the arithmetic your DL01.6 engine performs, written out in NumPy with the batch rules of the course: \(\mat{G}\T\mat{A}\) for a weight, \(\mat{G}\mat{W}\) for the layer below, column sums for a bias.

from sklearn.model_selection import train_test_split

X, y = make_moons(n_samples=200, noise=0.2, random_state=0)
X_tr, X_va, y_tr, y_va = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
rng = np.random.default_rng(0)                        # the generator of DL01.5.4 and DL01.6.4
sizes = [2, 16, 16, 2]
weights = [[rng.normal(0, np.sqrt(2 / n_in), (n_out, n_in)), np.zeros(n_out)]
           for n_in, n_out in zip(sizes[:-1], sizes[1:])]
idx = rng.permutation(len(y_tr))[:32]                 # the first batch of the first epoch
Xb, yb = X_tr[idx], y_tr[idx]

model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 16), nn.ReLU(), nn.Linear(16, 2)).double()
model.load_state_dict({f"{i}.{part}": torch.from_numpy(t)
                       for i, (W, b) in zip([0, 2, 4], weights) for part, t in (("weight", W), ("bias", b))})
opt = torch.optim.SGD(model.parameters(), lr=0.2)
opt.zero_grad()
first_loss = nn.CrossEntropyLoss()(model(torch.from_numpy(Xb)), torch.from_numpy(yb))
first_loss.backward()
opt.step()
print(f"first loss {first_loss.item():.3f}")

# The same step by hand, as the engine computes it.
A, cache = Xb, []
for l, (W, b) in enumerate(weights):
    Zl = A @ W.T + b
    cache.append((A, Zl))
    A = np.maximum(Zl, 0.0) if l < 2 else Zl
P = np.exp(A - A.max(axis=1, keepdims=True)); P /= P.sum(axis=1, keepdims=True)
G = (P - np.eye(2)[yb]) / len(yb)
for l in reversed(range(3)):
    A_in, Zl = cache[l]
    W, b = weights[l]
    dW, db = G.T @ A_in, G.sum(axis=0)
    if l > 0:
        G = (G @ W) * (cache[l - 1][1] > 0)
    weights[l] = [W - 0.2 * dW, b - 0.2 * db]

for (name, p), arr in zip(model.named_parameters(), [t for pair in weights for t in pair]):
    print(f"{name:9} largest difference {np.abs(p.detach().numpy() - arr).max():.1e}")
first loss 1.005
0.weight  largest difference 2.8e-17
0.bias    largest difference 1.4e-17
2.weight  largest difference 5.6e-17
2.bias    largest difference 6.9e-18
4.weight  largest difference 5.6e-17
4.bias    largest difference 0.0e+00

The first loss, about 1.005, is the first loss DL01.6.4 prints for the engine, and every one of the six tensors agrees with the engine’s arithmetic to rounding. PyTorch’s loss object, autograd and optimizer together did what your engine did; DL01.8 puts the five lines of this step in a loop.

Explore

This module’s interactive, Shapes and strides, lives in DL01.7.1. For the objects of this lesson, change lr in the running step to 1.0 and predict, from the worked example, what happens to the two hidden units and what the network’s logits become.

Pitfalls

Two panels plotting the loss for the true class against the logit margin of the true class, from minus 6 to 6. Left, marked with a cross and titled softmax, then the loss: a reddish-purple curve that falls only from about 1.31 to a dashed floor labelled never below 0.313, with the note flat: small gradients. Right, marked with a tick and titled logits straight in: a reddish-purple curve that falls towards 0 as the margin grows, with the point 1.313 at margin minus 1 marked.
DL01.7.4-F5

Wrong beside right. With an extra softmax the loss is squeezed between 0.313 and 1.313 for two classes, and its slope is small everywhere, so learning is slow and never finishes. Logits go straight in.

“My network ends in a softmax, so its outputs go into CrossEntropyLoss.”

nn.CrossEntropyLoss applies log-softmax itself. A softmax before it is applied twice: no error, a loss that cannot fall below \(-1 + {\ln(e + C - 1)}\), and gradients a fraction of the right ones. Return logits from forward, and apply torch.softmax only when you want to read probabilities. The same holds for a sigmoid before nn.BCEWithLogitsLoss.

“Labels are numbers, so a float tensor will do.”

nn.CrossEntropyLoss wants class indices as int64 of shape \((N,)\); a float tensor of shape \((N,)\) is refused. A float tensor of shape \((N, C)\) is accepted, but read as a probability distribution over classes for each example, so labels stored as floats by accident in that shape give wrong losses without an error. nn.BCEWithLogitsLoss is the opposite: its targets must be floats with the shape of the logits.

“The optimizer has a copy of my weights.”

It has the weights themselves. So make it after the model, from that model’s parameters(); if you later replace a layer (for example model[4] = nn.Linear(16, 3)), the optimizer still holds the old tensors and will never update the new layer. Make a new optimizer.

Check yourself

Recall

1. What inputs does nn.CrossEntropyLoss expect for a batch of \(N\) examples and \(C\) classes?

Show answer

Logits of shape \((N, C)\), not probabilities, and integer labels of dtype int64 and shape \((N,)\).

2. What is the default reduction of the loss objects, and what does it do?

Show answer

"mean": the per-example losses are averaged, so each example’s gradient carries a factor \(1/N\).

3. What does optimizer.step() do for plain SGD?

Show answer

For each parameter \(\theta\) it holds, \(\theta \leftarrow \theta - \eta\,\theta.\mathrm{grad}\), in place and outside the graph.

4. What does optimizer.zero_grad() do by default in PyTorch 2.15?

Show answer

It sets every parameter’s .grad to None (set_to_none=True) instead of filling it with zeros.

5. What is the smallest value two-class cross-entropy can take if a softmax is applied before nn.CrossEntropyLoss?

Show answer

\(-1 + {\ln(e + 1)} \approx 0.3133\), reached only when the probability of the true class is 1.

Exercises

1. ★ Compute by hand nn.BCEWithLogitsLoss() for logits \((0, 3)\) and targets \((1, 0)\), to three decimals.

Show solution

For \(z = 0\), \(y = 1\): \(\operatorname{softplus}(0) = \ln 2 \approx 0.693\). For \(z = 3\), \(y = 0\): \({\operatorname{softplus}(3)} = {\ln(1 + e^3)} \approx 3.049\). The mean is \({(0.693 + 3.049)/2} \approx 1.871\).

2. ★★ The same batch is trained once with reduction="mean" and once with reduction="sum", batch size 32, plain SGD. What learning rate with "sum" gives exactly the steps of lr=0.1 with "mean"? Why does this matter when the last batch of an epoch is shorter?

Show solution

The sum is 32 times the mean, so its gradient is 32 times larger, and \(\eta = 0.1/32 = 0.003125\) gives the same steps. But the factor is the batch size: on a short last batch of, say, 13 examples, "sum" makes steps \(13/32\) as large as a full batch would with that learning rate, while "mean" keeps the scale of the step independent of the batch size. DL01.9.3 returns to this.

3. ★★★ Show that with \(C\) classes and a softmax before nn.CrossEntropyLoss, the loss is at least \(-1 + {\ln(e + C - 1)}\), and check the value for \(C = 10\).

Show solution

With \(q_k = t\) and the other \(q_j\) summing to \(1 - t\), the loss is \(-t + {\ln\big(e^t + \sum_{j \ne k} e^{q_j}\big)}\). Every \(q_j \ge 0\), so each of the \(C - 1\) terms \(e^{q_j}\) is at least 1 and their sum is at least \(C - 1\). The logarithm increases, so the loss is at least \(g(t) = -t + {\ln(e^t + C - 1)}\). Its derivative is

\[ \begin{aligned} g'(t) &= -1 + \frac{e^t}{e^t + C - 1} \\ &= \frac{-(C - 1)}{e^t + C - 1} < 0, \end{aligned} \]

so \(g\) is smallest at \(t = 1\): \(g(1) = -1 + {\ln(e + C - 1)}\), reached by the one-hot \(\vect{q}\). For \(C = 10\) this is \(-1 + \ln(11.718) \approx 1.461\): the network on the digits could never report a loss below 1.461.

Summary card

A loss object turns logits and labels into one number; an optimizer holds the parameters and steps them.

The anchor figure again: logits and integer labels entering nn.CrossEntropyLoss, which applies log-softmax, picks the true class and takes the mean; and an SGD object with references to the orange parameters W1, b1, W2, b2, whose step and zero_grad act on them in place.
DL01.7.4-F1

Loss and optimizer as objects.

  • CrossEntropyLoss. Logits \((N, C)\), int64 labels \((N,)\), mean of \(-z_k + \ln\sum_j e^{z_j}\).
  • BCEWithLogitsLoss. Logits and float targets, mean of \({\operatorname{softplus}((1 - 2y)z)}\).
  • No softmax first. It floors the loss at \(-1 + {\ln(e + C - 1)}\).
  • SGD. step(): \(\theta \leftarrow \theta - \eta\,\theta.\mathrm{grad}\); zero_grad(): .grad to None.

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • PyTorch 2.15 documentation, torch.nn.CrossEntropyLoss: the input is expected to contain unnormalised logits; with class-index targets it is equivalent to LogSoftmax followed by NLLLoss; targets are class indices of dtype long with shape \((N)\), or class probabilities with the shape of the input; the default reduction is 'mean'.
  • PyTorch 2.15 documentation, torch.nn.BCEWithLogitsLoss: it combines a sigmoid and the binary cross-entropy in one class, which is more numerically stable by the log-sum-exp trick; the default reduction is 'mean'.
  • PyTorch 2.15 documentation, torch.optim.SGD: the signature with lr=0.001, momentum=0, dampening=0, weight_decay=0, nesterov=False, and the update \(\theta_t = \theta_{t-1} - \gamma g_t\) when momentum and weight decay are zero.
  • PyTorch 2.15 documentation, torch.optim.Optimizer.zero_grad: zero_grad(set_to_none=True) sets the gradients to None instead of zero, and an optimizer skips the step of a parameter whose gradient is None.
  • A. Zhang, Z. C. Lipton, M. Li and A. J. Smola, Dive into Deep Learning, section 4.5: the cross-entropy loss is passed unnormalised logits, and computing the softmax and its logarithm together avoids overflow and underflow (the LogSumExp trick).

Next. DL01.8, Anatomy of a training loop, puts zero_grad(), backward() and step() in a loop, and shows what goes wrong when their order changes.

This lesson unlocks

DL01.8 Anatomy of a training loop

Your progress is saved in this browser only. No account, no tracking.
Back to top