Skip to main content

A layer of neurons

Put several neurons side by side on the same input and their weight vectors stack into the rows of a matrix. A layer computes phi(Wx + b) for one example and phi(XW^T + 1b^T) for a batch. Learn the shapes, count the parameters and read the outputs as new features.
DL01.1.3L2 Intermediate25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: DL01.1.2 The perceptron and its learning rule M02.5 Matrix multiplication as composition

What you will be able to do
  • Stack the weight vectors of several neurons into a weight matrix, and compute a layer’s output for one example.
  • Compute a layer on a batch with rows as examples, and state the shape of every array.
  • Count the parameters of a layer as n_out (n_in + 1).
  • Read the outputs of a layer as new features of the data, and test whether a classifier can use them.

The question

The digit classifier at the end of this course reads 64 pixels and passes them to 32 neurons at once. Writing out 32 weighted sums of 64 terms each, for every image, would be hopeless by hand and slow in a Python loop. How do 32 neurons become one operation on a whole batch of images, and how many numbers do they learn?

From the last lesson

1. Write the perceptron’s update rule for \(\vect{w}\) and \(b\).

Show answer

\({\vect{w} \leftarrow \vect{w} + \eta(y - \hat{y})\vect{x}}\) and \({b \leftarrow b + \eta(y - \hat{y})}\).

2. What happens when the perceptron is run on XOR?

Show answer

It never settles: it makes mistakes in every epoch and cycles through the same weights.

Intuition

Take several neurons and give them all the same input. Each has its own weights and its own bias, so each computes its own weighted sum and its own output. Side by side they form a layer.

Left: two blue input nodes x1 and x2, each joined by thin orange edges to three vermillion neuron nodes marked phi, whose outputs are a1, a2 and a3; the two edges into the middle neuron are thick and labelled w21 and w22, and that neuron is filled yellow. Labels read input and 3 neurons. Right: an orange grid labelled W, 3 by 2, with entries w11, w12 in the first row, w21, w22 in the second and w31, w32 in the third; the second row is filled yellow and marked neuron 2. Beside it a column labelled b with entries b1, b2 and b3, the second also yellow. Under the grid: one row per neuron, one column per input, and the formula a equals phi of W x plus b.
DL01.1.3-F1

A layer of three neurons on two inputs. The weights of neuron \(i\) fill row \(i\) of the weight matrix \(\mat{W}\), and its bias is entry \(i\) of \(\vect{b}\). Neuron 2 is picked out in both views.

The picture on the right is the whole lesson. Each neuron is a row of \(\mat{W}\). The product \(\mat{W}\vect{x}\) takes the dot product of each row with \(\vect{x}\), so it computes every neuron’s weighted sum at once. Add the bias vector, apply the activation to every entry, and the layer is done.

Build it up

Step 1. Three neurons, written out. With inputs \(\vect{x} = (x_1, x_2)\), three neurons compute

\[ \begin{aligned} z_1 &= w_{11}x_1 + w_{12}x_2 + b_1 , \\ z_2 &= w_{21}x_1 + w_{22}x_2 + b_2 , \\ z_3 &= w_{31}x_1 + w_{32}x_2 + b_3 , \end{aligned} \]

and then \({a_i = \phi(z_i)}\). The weight \(w_{ij}\) is the weight of neuron \(i\) on input \(j\).

Step 2. Stack the rows. Put the three weight vectors into the rows of a \(3 \times 2\) matrix and the biases into a vector:

\[ \mat{W} = \begin{bmatrix} w_{11} & w_{12} \\ w_{21} & w_{22} \\ w_{31} & w_{32} \end{bmatrix}, \quad \vect{b} = \begin{bmatrix} b_1 \\ b_2 \\ b_3 \end{bmatrix} . \]

Row \(i\) of \({\mat{W}\vect{x}}\) is the dot product of row \(i\) of \(\mat{W}\) with \(\vect{x}\) (M02.5), which is neuron \(i\)’s weighted sum without its bias. So the three lines of Step 1 are one line: \({\vect{z} = \mat{W}\vect{x} + \vect{b}}\).

Step 3. Activate entry by entry. The activation acts on each entry on its own: \({\vect{a} = \phi(\vect{z}) = (\phi(z_1), \phi(z_2), \phi(z_3))}\). The outputs \(\vect{a}\) are called the layer’s activations.

Step 4. Each neuron draws a line. Each row of \(\mat{W}\), with its bias, defines a line \({z_i = 0}\) in the input plane, exactly as the perceptron did. A layer of three neurons cuts the plane with three lines. With the ReLU, a point’s activation from a neuron is 0 on that neuron’s off side and grows with the distance beyond its line on the on side.

The input plane from minus 2.6 to 2.6 with a quiet grid. Three vermillion lines: neuron 1, x1 plus x2 equals 0, solid, running from upper left to lower right through the origin; neuron 2, x1 minus x2 equals 0, dashed, through the origin the other way; neuron 3, 2 x2 minus 1 equals 0, dotted, horizontal at height 0.5. A short orange arrow numbered 1, 2 or 3 leaves each line towards the side where that neuron is on. Two blue points are marked, (2, 1) and (1, 2). Text beside the plane lists the three lines, says arrows point to the side where the neuron is on, and gives the features: of (2, 1), (3, 1, 1); of (1, 2), (3, 0, 3).
DL01.1.3-F5

The layer of the worked example seen in the input plane. Each neuron has its own line, and its weight vector points to the side where its weighted sum is positive. The point \((1, 2)\) is on the off side of neuron 2, so its second feature is 0.

Step 5. A batch. In practice a layer processes many examples at once. Stack them as the rows of a matrix \(\mat{X}\), one row per example, which is how data tables and NumPy arrays are laid out. The next section shows that the whole batch is then one matrix product.

Five blocks of cells in a row, joined by operators. A blue block X, 4 rows by 2 columns, shape n by n-in; a dot; an orange block W transpose, 2 by 3, shape n-in by n-out; a plus; an orange block 1 b transpose, 4 by 3, whose first row is solid and whose other three rows are faint dashed copies, shape n by n-out; an equals sign; a vermillion block Z, 4 by 3; an arrow labelled phi; a vermillion block A, 4 by 3. A note underneath: each row of X is one example; b transpose is copied to every row.
DL01.1.3-F2

The shapes of a layer on a batch of \({n = 4}\) examples with \({n_{\text{in}} = 2}\) inputs and \({n_{\text{out}} = 3}\) neurons. The bias row is copied to every example.

The formal statement

NoteDefinition: a fully connected layer

A layer with \(n_{\text{in}}\) inputs and \(n_{\text{out}}\) neurons has a weight matrix \(\cparam{\mat{W}}\) of shape \({n_{\text{out}} \times n_{\text{in}}}\), a bias vector \(\cparam{\vect{b}}\) of length \(n_{\text{out}}\) and an activation \(\phi\) applied entry by entry. For one input \(\cdata{\vect{x}}\),

\[ \cmodel{\vect{a}} = \phi(\cparam{\mat{W}}\cdata{\vect{x}} + \cparam{\vect{b}}) . \]

For a batch \(\cdata{\mat{X}}\) of shape \({n \times n_{\text{in}}}\), one example per row,

\[ \begin{aligned} \mat{Z} &= \cdata{\mat{X}}\cparam{\mat{W}}\T + \vect{1}\cparam{\vect{b}}\T , \\ \cmodel{\mat{A}} &= \phi(\mat{Z}) , \end{aligned} \]

where \(\vect{1}\) is a column of \(n\) ones, so that \({\vect{1}\vect{b}\T}\) repeats \(\vect{b}\T\) in every row.

Why the batch formula has a transpose. Row \(i\) of \(\mat{X}\) is \(\vect{x}_i\T\), and the layer’s weighted sums for that example are the column \({\mat{W}\vect{x}_i + \vect{b}}\). To store them as row \(i\) of \(\mat{Z}\), transpose, using \({(\mat{A}\mat{B})\T = \mat{B}\T\mat{A}\T}\):

\[ \begin{aligned} (\mat{W}\vect{x}_i + \vect{b})\T &= (\mat{W}\vect{x}_i)\T + \vect{b}\T \\ &= \vect{x}_i\T\mat{W}\T + \vect{b}\T . \end{aligned} \]

Stacking these rows for \({i = 1, \dots, n}\) stacks the \(\vect{x}_i\T\) into \(\mat{X}\), and gives \({\mat{Z} = \mat{X}\mat{W}\T + \vect{1}\vect{b}\T}\). In NumPy the term \({\vect{1}\vect{b}\T}\) is written + b: broadcasting copies the vector to every row.

The shapes.

Array Shape Holds
\(\mat{X}\) \({n \times n_{\text{in}}}\) one example per row
\(\mat{W}\) \({n_{\text{out}} \times n_{\text{in}}}\) one neuron per row
\(\vect{b}\) \(n_{\text{out}}\) one bias per neuron
\(\mat{Z}\), \(\mat{A}\) \({n \times n_{\text{out}}}\) one example per row

\(\mat{W}\) keeps the shape (out, in), the shape in which PyTorch’s nn.Linear stores its weight (DL01.7.3). M05.7.3, and scikit-learn’s MLPClassifier (DL01.2.2), write a layer as \(\mat{X}\mat{W}\) with \(\mat{W}\) of shape (in, out): the same layer, with the matrix stored transposed.

Counting parameters. Each neuron has \(n_{\text{in}}\) weights and one bias, so a layer has

\[ n_{\text{out}}\,n_{\text{in}} + n_{\text{out}} = n_{\text{out}}\,(n_{\text{in}} + 1) \]

parameters. The digit network’s hidden layer has \({32 \cdot (64 + 1) = 2{,}080}\); its output layer, ten neurons on 32 inputs, has \({10 \cdot 33 = 330}\); together \({2{,}410}\), the number that DL01.7.3 checks in PyTorch.

Outputs as features. A layer replaces each example’s \(n_{\text{in}}\) numbers by \(n_{\text{out}}\) new ones. Any model placed after the layer sees only these new numbers, so the layer decides which features of the data the rest of the network can use. With fixed random weights it is a random feature map; once the weights are trained (DL01.2 onwards), the network chooses its own features.

Worked example

A layer of three ReLU neurons on a batch of two examples. Take

\[ \mat{W} = \begin{bmatrix} 1 & 1 \\ 1 & -1 \\ 0 & 2 \end{bmatrix}, \quad \vect{b} = \begin{bmatrix} 0 \\ 0 \\ -1 \end{bmatrix}, \]

\[ \mat{X} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} . \]

The batch holds the examples \((2, 1)\) and \((1, 2)\). Each entry of \({\mat{X}\mat{W}\T}\) is an example (a row of \(\mat{X}\)) dotted with a neuron (a row of \(\mat{W}\)). For the first example:

  • neuron 1: \({2 \cdot 1 + 1 \cdot 1 + 0 = 3}\);
  • neuron 2: \({2 \cdot 1 + 1 \cdot (-1) + 0 = 1}\);
  • neuron 3: \({2 \cdot 0 + 1 \cdot 2 - 1 = 1}\).

For the second: \({1 + 2 = 3}\), \({1 - 2 = -1}\) and \({0 + 4 - 1 = 3}\). So

\[ \mat{Z} = \begin{bmatrix} 3 & 1 & 1 \\ 3 & -1 & 3 \end{bmatrix}, \]

\[ \mat{A} = \text{ReLU}(\mat{Z}) = \begin{bmatrix} 3 & 1 & 1 \\ 3 & 0 & 3 \end{bmatrix} . \]

A trace of the layer on the batch. Top row: a blue 2 by 2 grid X with rows 2, 1 and 1, 2; a dot; an orange 2 by 3 grid W transpose with rows 1, 1, 0 and 1, minus 1, 2; a plus; an orange row b transpose holding 0, 0, minus 1, noted added to each row; an equals sign; a vermillion 2 by 3 grid Z with rows 3, 1, 1 and 3, minus 1, 3, the minus 1 filled yellow. Under it, three lines of arithmetic: row 1 column 1, 2 times 1 plus 1 times 1 plus 0 equals 3; row 2 column 2, 1 times 1 plus 2 times minus 1 plus 0 equals minus 1; row 1 column 3, 2 times 0 plus 1 times 2 minus 1 equals 1. Bottom right: Z again, an arrow labelled ReLU, and the grid A with rows 3, 1, 1 and 3, 0, 3, where the yellow cell now holds 0; a note says max of 0 and minus 1 equals 0, neuron 2 is off for example 2.
DL01.1.3-F3

The worked example as a trace. Every entry of \(\mat{Z}\) is one example dotted with one neuron, plus that neuron’s bias; ReLU then zeroes the single negative entry.

Read the rows of \(\mat{A}\) as features. The example \((2, 1)\) is described by \((3, 1, 1)\) and \((1, 2)\) by \((3, 0, 3)\): neuron 1 gives both the same value, neuron 2 is on for one and off for the other, and neuron 3 tells them apart by how high they sit. The layer has \({3 \cdot (2 + 1) = 9}\) parameters.

Code

A layer as a NumPy function, with every shape asserted, so that a transposed matrix fails loudly instead of computing the wrong thing.

import numpy as np

def relu(z):
    return np.maximum(z, 0.0)

def layer(X, W, b, phi):
    """A fully connected layer on a batch. X is (n, n_in), W is (n_out, n_in), b is (n_out,)."""
    n, n_in = X.shape
    assert b.ndim == 1, f"b must be a vector of length n_out, got shape {b.shape}"
    n_out = b.shape[0]
    assert W.shape == (n_out, n_in), f"W must be (n_out, n_in) = ({n_out}, {n_in}), got {W.shape}"
    Z = X @ W.T + b                       # broadcasting adds b to every row: the 1 b^T of the formula
    assert Z.shape == (n, n_out)
    return phi(Z)

def n_params(n_in, n_out):
    return n_out * (n_in + 1)

W = np.array([[1.0, 1.0], [1.0, -1.0], [0.0, 2.0]])
b = np.array([0.0, 0.0, -1.0])
X = np.array([[2.0, 1.0], [1.0, 2.0]])
A = layer(X, W, b, relu)
print(A)
print("parameters:", n_params(2, 3), " digit network:", n_params(64, 32) + n_params(32, 10))
[[3. 1. 1.]
 [3. 0. 3.]]
parameters: 9  digit network: 2410

The batch formula agrees with three neurons computed one at a time in a loop, and the shape checks catch the classic slip of multiplying by \(\mat{W}\) instead of \(\mat{W}\T\).

loop = np.array([[relu(W[i] @ x + b[i]) for i in range(3)] for x in X])
print("loop equals batch:", np.array_equal(loop, A))
try:
    layer(X, W.T, b, relu)                # W stored as (in, out) by mistake
except AssertionError as error:
    print("caught:", error)
loop equals batch: True
caught: W must be (n_out, n_in) = (3, 2), got (2, 3)

Now the outputs as features. Take the two moons of scikit-learn (200 points, the course’s running two-class set) and a layer of 50 ReLU neurons with random weights that are never trained. A logistic regression on the raw two coordinates can only draw a straight boundary; the same model on the 50 features of the layer does much better.

from sklearn.datasets import make_moons
from sklearn.linear_model import LogisticRegression

X_moons, y_moons = make_moons(n_samples=200, noise=0.2, random_state=0)
rng = np.random.default_rng(0)
W_rand = rng.normal(size=(50, 2))
b_rand = rng.normal(size=50)
F = layer(X_moons, W_rand, b_rand, relu)                  # (200, 2) in, (200, 50) out

raw = LogisticRegression().fit(X_moons, y_moons).score(X_moons, y_moons)
feat = LogisticRegression().fit(F, y_moons).score(F, y_moons)
print("feature array:", F.shape, " layer parameters:", n_params(2, 50))
print(f"training accuracy on the raw inputs: {raw:.3f}   on the layer's features: {feat:.3f}")
feature array: (200, 50)  layer parameters: 150
training accuracy on the raw inputs: 0.840   on the layer's features: 0.945

The accuracies are on the training points (with scikit-learn’s default penalty, C=1), so they measure what each set of features can express, not how well it generalises; ML01.2 explains the difference. The point is the jump from about 84 to about 94 percent without training the layer at all: new features made a linear model stronger. Choosing those features well is what training a network does.

Explore

The module’s interactive is in lesson 2. It shows one neuron, one row of \(\mat{W}\): each setting of its sliders is one line of the picture in Step 4.

Pitfalls

Two panels, each a product of a blue row x transpose equal to 2, 1 with an orange 2 by 2 grid. Left, marked with a cross and titled X W, runs but wrong: the grid W with rows 1, 2 and 0, 1, and the result 2, 5; the note says first entry 2 times 1 plus 1 times 0 equals 2, it mixes the first weight of each neuron, not one neuron. Right, marked with a tick and titled X W transpose: the grid W transpose with rows 1, 0 and 2, 1, and the result 4, 1; the note says first entry 2 times 1 plus 1 times 2 equals 4, neuron 1's weights (1, 2) dotted with the input.
DL01.1.3-F4

Wrong beside right, for a layer with neurons \((1, 2)\) and \((0, 1)\) on the input \((2, 1)\). With a square \(\mat{W}\) both products run, and only one is the layer.

“If the matrix product runs, the shapes are right.”

When \({n_{\text{in}} = n_{\text{out}}}\), \(\mat{W}\) is square and \(\mat{X}\mat{W}\) runs just as happily as \(\mat{X}\mat{W}\T\). It dots each example with the columns of \(\mat{W}\), which mix the weights of different neurons. Write the shapes down, or assert them, as the layer function does.

“The bias is added once to the whole batch.”

It is added to every row: each example gets every neuron’s bias. In NumPy, X @ W.T + b with b of shape (n_out,) does this. Writing b[:, None] instead, a column, broadcasts along the wrong axis; when \({n = n_{\text{out}}}\) it runs silently and adds bias \(i\) to example \(i\).

“A layer of several neurons draws a curved boundary.”

Each neuron still draws one straight line. A layer only produces features; a boundary appears when something reads them, such as another neuron. Lesson 4 shows that without an activation between them, even that does not bend the boundary.

Check yourself

Recall

1. In the weight matrix of a layer, what does row \(i\) hold, and what does column \(j\) hold?

Show answer

Row \(i\) holds the weights of neuron \(i\); column \(j\) holds every neuron’s weight on input \(j\).

2. Write a layer’s output for one example and for a batch with rows as examples.

Show answer

\({\vect{a} = \phi(\mat{W}\vect{x} + \vect{b})}\) and \({\mat{A} = \phi(\mat{X}\mat{W}\T + \vect{1}\vect{b}\T)}\).

3. What are the shapes of \(\mat{X}\), \(\mat{W}\) and \(\mat{Z}\) for a batch of \(n\) examples?

Show answer

\({n \times n_{\text{in}}}\), \({n_{\text{out}} \times n_{\text{in}}}\) and \({n \times n_{\text{out}}}\).

4. How many parameters does a layer with \(n_{\text{in}}\) inputs and \(n_{\text{out}}\) neurons have?

Show answer

\({n_{\text{out}}(n_{\text{in}} + 1)}\): a weight per neuron per input, plus a bias per neuron.

5. In the moons experiment, what changed between the 84 and the 94 percent accuracy?

Show answer

Only the inputs of the logistic regression: the raw two coordinates, or the 50 features of an untrained random ReLU layer.

Exercises

1. ★ A network reads \(28 \times 28 = 784\) pixels into a layer of 128 neurons, then a layer of 10. How many parameters does each layer have, and how many in all?

Show solution

\({128 \cdot (784 + 1) = 100{,}480}\) and \({10 \cdot (128 + 1) = 1{,}290}\), so \({101{,}770}\) in all.

2. ★★ For the layer of the worked example, compute the activations of the inputs \((-1, 1)\) and \((0, 0)\). What is the shape of \(\mat{Z}\) for a batch of 5 examples?

Show solution

For \((-1, 1)\):

\[ \begin{aligned} \vect{z} &= (-1 + 1,\; -1 - 1,\; 0 + 2 - 1) \\ &= (0, -2, 1) , \end{aligned} \]

so \({\vect{a} = (0, 0, 1)}\). For \((0, 0)\): \({\vect{z} = \vect{b} = (0, 0, -1)}\), so \({\vect{a} = (0, 0, 0)}\). For 5 examples, \(\mat{Z}\) is \({5 \times 3}\).

3. ★★★ Show that \({\mat{X}\mat{W}\T + \vect{1}\vect{b}\T = \tilde{\mat{X}}\tilde{\mat{W}}\T}\), where \(\tilde{\mat{X}}\) is \(\mat{X}\) with a column of ones appended and \(\tilde{\mat{W}}\) is \(\mat{W}\) with \(\vect{b}\) appended as a last column. Which lesson used the same trick?

Show solution

Write \({\tilde{\mat{X}} = [\,\mat{X} \;\; \vect{1}\,]}\) and \({\tilde{\mat{W}} = [\,\mat{W} \;\; \vect{b}\,]}\), so \({\tilde{\mat{W}}\T}\) stacks \(\mat{W}\T\) on top of the row \(\vect{b}\T\). Multiplying blocks,

\[ \begin{aligned} \tilde{\mat{X}}\tilde{\mat{W}}\T &= [\,\mat{X} \;\; \vect{1}\,] \begin{bmatrix} \mat{W}\T \\ \vect{b}\T \end{bmatrix} \\ &= \mat{X}\mat{W}\T + \vect{1}\vect{b}\T . \end{aligned} \]

Lesson 2 used the same trick for one neuron, folding the bias into the weights as \({\vect{\theta} = (\vect{w}, b)}\) with inputs \({\tilde{\vect{x}} = (\vect{x}, 1)}\).

Summary card

A layer is many neurons on one input: their weights are the rows of W.

The anchor figure again: two inputs joined to three neurons, and the 3 by 2 weight matrix W whose second row, the weights of neuron 2, is highlighted beside the bias vector b.
DL01.1.3-F1

One row per neuron.

  • One example. \({\vect{a} = \phi(\mat{W}\vect{x} + \vect{b})}\), with \(\mat{W}\) of shape (out, in).
  • A batch. \({\mat{A} = \phi(\mat{X}\mat{W}\T + \vect{1}\vect{b}\T)}\), one example per row.
  • Parameters. \({n_{\text{out}}(n_{\text{in}} + 1)}\); the digit network has \({2{,}080 + 330 = 2{,}410}\).
  • Features. The rows of \(\mat{A}\) are new features of the examples.

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • Stanford CS231n, Neural Networks Part 1: Setting up the Architecture, sections “Layer-wise organization” and “Example feed-forward computation”: “The forward pass of a fully-connected layer corresponds to one matrix multiplication followed by a bias offset”, and a network with layers of 4 and 2 neurons on 3 inputs has \({[3 \times 4] + [4 \times 2] = 20}\) weights and \({4 + 2 = 6}\) biases, 26 parameters in all.
  • Simon J. D. Prince, Understanding Deep Learning (MIT Press, 2023), Section 4.4, “Matrix notation”: a layer as an activation applied to a bias vector plus a weight matrix times the previous layer, with the weight matrix of size (units out) by (units in).
  • PyTorch 2.15 documentation, torch.nn.Linear: computes \({y = xA\T + b}\), with the weight of shape (out_features, in_features) and the bias of shape (out_features).

Next. DL01.1.4 What one layer cannot do asks whether a single layer, or a stack of layers with nothing between them, can classify XOR.

This lesson unlocks

DL01.1.4 What one layer cannot do

Your progress is saved in this browser only. No account, no tracking.
Back to top