Four honest ways to measure how big a matrix is: the length of its entries, its largest stretch, its largest column sum and its largest row sum. Derive each from the singular values or the unit shapes, and prove that the stretch of a product is at most the product of the stretches.
M03.8.1L2 Intermediate25 minIn review (what this means)
The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.
State the three properties a matrix norm must have, and check them for the Frobenius norm.
Compute the Frobenius, 2-, 1-, max- and nuclear norms of a small matrix by hand, and with numpy.linalg.norm.
Derive that the 2-norm is the largest singular value and that the Frobenius norm is the square root of the sum of the squared singular values.
Prove the submultiplicative rule for induced norms and use it to bound repeated products.
The question
A layer of a neural network multiplies its input by a weight matrix \(\mat{W}\). If the input changes a little, how much can the output change? To answer, we need one number for “how big” \(\mat{W}\) is. A matrix holds many numbers, so which one number is the honest answer?
Intuition
There are two natural ways to look at a matrix, and each gives a way to measure it.
As a list of numbers. A 2 by 2 matrix holds four numbers, so it is a vector with four entries written in a square. Its length, in the ordinary sense of lesson M02.3.2, is one measure of size.
As a map. A matrix moves vectors. Feed it every vector of length 1 and look at how long the outputs are. The longest one tells you the most the matrix can magnify anything. That is the measure the question asks for.
M03.8.1-F1
The matrix \(\mat{M}\) with rows \({(3, 0)}\) and \({(4, 5)}\), the running SVD matrix of M03.5, sends the unit circle to an ellipse. Its longest radius, \({\sigma_1 \approx 6.71}\), is the largest stretch.
We will meet four measures in all. The Frobenius norm treats the matrix as a list. The 2-norm is the longest arrow in the picture. Two more come from measuring the arrows with the other rulers of M02.3.2, the 1-norm and the max-norm. They all agree on what “zero” means, and they all grow when the matrix grows, but they give different numbers.
Build it up
Step 1. The entries as one long vector. Put the entries of \(\mat{M}\) in a row, \({(3, 0, 4, 5)}\), and take the usual length: \({\sqrt{9 + 0 + 16 + 25} = \sqrt{50} \approx 7.07}\). This is the Frobenius norm, \({\norm{\mat{M}}_F}\). It ignores the shape completely: the same four numbers in another order would give the same answer.
Step 2. The matrix as a stretcher. Take a unit vector \(\vect{x}\) pointing at angle \(\theta\) and measure \({\norm{\mat{M}\vect{x}}}\). As \(\theta\) turns, the length rises and falls. The stretch factor of a non-zero vector is \({\norm{\mat{M}\vect{x}} / \norm{\vect{x}}}\); for a unit vector it is just the length of the image.
M03.8.1-F2
The stretch of \(\mat{M}\) in every direction. The top of the curve is the 2-norm, \(\sigma_1\); the bottom is \(\sigma_2\). The Frobenius norm sits above the curve.
Step 3. Keep the largest. The top of the curve is the 2-norm, \({\norm{\mat{M}}_2 \approx 6.71}\), reached at \(45°\). The bottom, \(2.24\) at \(135°\), will matter in the next lesson. Both are singular values from M03.5: the peak is at the right singular vector \(\vect{v}_1\) and the low at \(\vect{v}_2\).
Step 4. Change the ruler. Nothing forces us to measure arrows with the ordinary length. Use the 1-norm on both inputs and outputs, and the largest stretch becomes the matrix 1-norm. Use the max-norm, and it becomes the matrix max-norm. Each is read off a picture of the unit shape and its image, as the figure in the formal section shows.
The formal statement
NoteDefinition: matrix norm
A matrix norm gives every \(m\) by \(n\) matrix a number \({\norm{\mat{A}}}\) such that, for all matrices \(\mat{A}, \mat{B}\) of that shape and every number \(c\),
\({\norm{\mat{A}} \ge 0}\), with equality only for the zero matrix;
These are the same three properties that made a vector norm in M02.3.2. The Frobenius norm has them because it is a vector norm, applied to the list of entries. An induced norm has them because each stretch \({\norm{\mat{A}\vect{x}}}\) has them, and taking the largest keeps them. By its definition an induced norm also gives, for every \(\vect{x}\),
The Frobenius norm through the trace. Entry \({(j, j)}\) of \(\mat{A}\T\mat{A}\) is column \(j\) dotted with itself, \(\sum_i a_{ij}^2\). Adding the diagonal adds every squared entry once:
The trace is the sum of the eigenvalues (M03.1.3), and the eigenvalues of \(\mat{A}\T\mat{A} = \mat{V}\mat{\Sigma}^2\mat{V}\T\) are the squared singular values (M03.5.2). So
The right side is the Rayleigh quotient of the symmetric matrix \(\mat{A}\T\mat{A}\) (M03.3.4). Its largest value is the largest eigenvalue, \(\sigma_1^2\), reached at the top eigenvector \(\vect{v}_1\). Take square roots: \({\norm{\mat{A}}_2 = \sigma_1}\). In the same way the smallest stretch of a matrix with \(n\) columns is \(\sigma_n\).
Comparing the two formulas, \({\sigma_1^2 \le \sum \sigma_i^2 \le r\,\sigma_1^2}\), so
The 1-norm is the largest column sum. Let \({c_j = \sum_i \abs{a_{ij}}}\) be the sum of sizes in column \(j\), and \(c_{\max}\) the largest. For any \(\vect{x}\), the triangle inequality for numbers gives
The third line swaps the order of the two sums, and the last replaces each \(c_j\) by the largest. So no stretch exceeds \(c_{\max}\), and the basis vector \(\vect{e}_j\) of the best column reaches it: \({\norm{\vect{e}_j}_1 = 1}\) and \({\norm{\mat{A}\vect{e}_j}_1 = c_j}\). Hence \({\norm{\mat{A}}_1 = c_{\max}}\). The same kind of argument (exercise 2) shows that the max-norm \({\norm{\mat{A}}_\infty}\) is the largest row sum.
M03.8.1-F3
The unit shapes of M02.3.2 and their images. The farthest image corner, measured with the same ruler, gives the norm: a column for the 1-norm, a row sum for the max-norm.
Products. For any induced norm, apply the bound \({\norm{\mat{A}\vect{y}}} \le {\norm{\mat{A}}\norm{\vect{y}}}\) twice, first with \({\vect{y} = \mat{B}\vect{x}}\):
Divide by \({\norm{\vect{x}}}\) and take the largest over \(\vect{x}\): \({\norm{\mat{A}\mat{B}}} \le {\norm{\mat{A}}\,\norm{\mat{B}}}\). This is the submultiplicative rule. Repeating it gives \({\norm{\mat{A}^k} \le \norm{\mat{A}}^k}\): a stack of layers can magnify at most by the product of their norms.
One more: the nuclear norm. The sum of the singular values, \({\norm{\mat{A}}_* = \sigma_1 + \cdots + \sigma_r}\), is also a matrix norm. It is the 1-norm of the list of singular values, as the Frobenius norm is their 2-norm.
Go deeper: the Frobenius norm is submultiplicative too
Entry \({(i, k)}\) of \(\mat{A}\mat{B}\) is row \(i\) of \(\mat{A}\) dotted with column \(k\) of \(\mat{B}\). By the Cauchy-Schwarz inequality its square is at most the squared length of that row times the squared length of that column. Add over all \(i\) and \(k\): the right side becomes \({\big(\sum_i \norm{\text{row}_i}^2\big)\big(\sum_k \norm{\text{col}_k}^2\big)} = {\norm{\mat{A}}_F^2\,\norm{\mat{B}}_F^2}\). So \({\norm{\mat{A}\mat{B}}_F} \le {\norm{\mat{A}}_F\,\norm{\mat{B}}_F}\), although the Frobenius norm is not an induced norm (see the pitfalls).
Worked example
Find five norms of \(\mat{M}\), the matrix with rows \({(3, 0)}\) and \({(4, 5)}\).
Frobenius. Square and add the entries: \({9 + 0 + 16 + 25 = 50}\), so \({\norm{\mat{M}}_F = \sqrt{50} \approx 7.07}\).
Its trace is 50 and its determinant is \({625 - 400 = 225}\), so its eigenvalues solve \({\lambda^2 - 50\lambda + 225 = 0}\), giving \(45\) and \(5\). The singular values are \({\sigma_1 = \sqrt{45} = 3\sqrt{5} \approx 6.71}\) and \({\sigma_2 = \sqrt{5} \approx 2.24}\), so \({\norm{\mat{M}}_2 \approx 6.71}\). Check the Frobenius norm: \({45 + 5 = 50}\).
1-norm. The column sums of sizes are \({3 + 4 = 7}\) and \({0 + 5 = 5}\), so \({\norm{\mat{M}}_1 = 7}\).
Max-norm. The row sums are \({3 + 0 = 3}\) and \({4 + 5 = 9}\), so \({\norm{\mat{M}}_\infty = 9}\).
M03.8.1-F4
Five norms of one matrix, each from a few lines of arithmetic. The 1- and max-norms come straight from the column and row sums; the 2- and nuclear norms need the singular values.
Five honest answers, from 6.71 to 9. The order fits the bounds: \({6.71 \le 7.07 \le \sqrt{2}\cdot 6.71 \approx 9.49}\). The check on the stretch curve: the vector \({\vect{v}_1 = (1, 1)/\sqrt{2}}\) goes to \({(3, 9)/\sqrt{2}}\), of length \({\sqrt{90}/\sqrt{2} = \sqrt{45}}\), the top of the curve.
Code
Each norm from scratch, with nothing but sums, square roots and the eigenvalues of a symmetric matrix.
import numpy as npM = np.array([[3.0, 0.0], [4.0, 5.0]])def frobenius(A):return np.sqrt((A **2).sum())def two_norm(A):"""The largest singular value: the square root of the top eigenvalue of A^T A."""return np.sqrt(np.linalg.eigvalsh(A.T @ A)[-1])def one_norm(A):return np.abs(A).sum(axis=0).max() # largest column sumdef max_norm(A):return np.abs(A).sum(axis=1).max() # largest row sumdef nuclear(A):return np.sqrt(np.clip(np.linalg.eigvalsh(A.T @ A), 0, None)).sum()for name, f in [("Frobenius", frobenius), ("2-norm", two_norm), ("1-norm", one_norm), ("max-norm", max_norm), ("nuclear", nuclear)]:print(f"{name:10s}{f(M):.4f}")# The 2-norm as the top of the stretch curve, by brute force over 3600 directions.theta = np.linspace(0, np.pi, 3600, endpoint=False)units = np.stack([np.cos(theta), np.sin(theta)])stretch = np.linalg.norm(M @ units, axis=0)print("largest stretch found:", round(stretch.max(), 4), "at", round(np.degrees(theta[stretch.argmax()]), 1), "degrees")
Frobenius 7.0711
2-norm 6.7082
1-norm 7.0000
max-norm 9.0000
nuclear 8.9443
largest stretch found: 6.7082 at 45.0 degrees
NumPy computes all five with one function. For a two-dimensional array, ord chooses the norm; leaving it out gives the Frobenius norm, not the 2-norm.
for ord_ in ["fro", 2, 1, np.inf, "nuc", None]:print(f"ord={ord_!s:5s}{np.linalg.norm(M, ord_):.4f}")# The submultiplicative rule on a product, and the bound for powers.K = np.array([[2.0, 3.0], [1.0, 2.0]])lhs, rhs = np.linalg.norm(M @ K, 2), np.linalg.norm(M, 2) * np.linalg.norm(K, 2)print("||MK|| =", round(lhs, 3), "<= ||M|| ||K|| =", round(rhs, 3))
This module’s interactive, the condition explorer, lives in the next lesson. There you drag the columns of a matrix and watch \(\sigma_1\) and \(\sigma_2\), the top and the bottom of the stretch curve, change together.
Pitfalls
M03.8.1-F5
Wrong beside right. The shear’s only eigenvalue is 1, yet it stretches some vectors by the golden ratio \({\varphi = (1 + \sqrt5)/2 \approx 1.62}\).
“The largest eigenvalue tells you how much a matrix can stretch.”
Only for a symmetric matrix. Eigenvectors are the directions that stay on their line, not the directions that grow most. The shear has the single eigenvalue 1 and stretches by \({\varphi \approx 1.62}\); the running matrix with rows \({(4, 1)}\) and \({(2, 3)}\) has eigenvalues 5 and 2 and a 2-norm of \({\sqrt{15 + 5\sqrt5} \approx 5.12}\). What is always true, for any induced norm, is that the spectral radius\(\rho(\mat{A})\), the largest size \({\abs{\lambda}}\) of an eigenvalue, is a lower bound: \({\rho(\mat{A}) \le \norm{\mat{A}}}\) (exercise 3).
“The Frobenius norm and the 2-norm are the same thing.”
They agree only when the rank is at most 1, since otherwise \(\sigma_2^2\) adds to the sum. The identity shows the gap: every induced norm gives \({\norm{\mat{I}} = 1}\), since \(\mat{I}\) stretches nothing, but \({\norm{\mat{I}_n}_F = \sqrt{n}}\). That is also why the Frobenius norm cannot be an induced norm. In code, np.linalg.norm(W) on a matrix returns the Frobenius norm; ask for ord=2 when you mean the largest stretch.
“Weight decay limits how much a layer can amplify.”
It does, but loosely. Weight decay penalises the sum of squared weights, which is \({\norm{\mat{W}}_F^2}\), the norm the Deep Learning book calls the most common way to measure a matrix there. Since \({\norm{\mat{W}}_2 \le \norm{\mat{W}}_F}\), a small Frobenius norm caps the stretch. A large one does not mean a large stretch: a big matrix of small, spread-out singular values can have a large Frobenius norm and a modest 2-norm.
Check yourself
Recall
1. Which three properties must a matrix norm have?
Show answer
It is never negative and is zero only for the zero matrix; scaling the matrix by \(c\) scales the norm by \({\abs{c}}\); and the norm of a sum is at most the sum of the norms.
2. How do the singular values give the Frobenius norm and the 2-norm?
3. How do you read \({\norm{\mat{A}}_1}\) and \({\norm{\mat{A}}_\infty}\) off the entries?
Show answer
Add the sizes of the entries in each column and take the largest for the 1-norm; do the same with rows for the max-norm.
4. What is the submultiplicative rule, and for which norms did this lesson prove it?
Show answer
\({\norm{\mat{A}\mat{B}}} \le {\norm{\mat{A}}\,\norm{\mat{B}}}\). It was proved for every induced norm (the 2-, 1- and max-norms), and in the deeper note for the Frobenius norm.
5. What does np.linalg.norm(A) return for a two-dimensional array?
Show answer
The Frobenius norm. The 2-norm needs ord=2.
Exercises
1.★ The running rectangular matrix \(\mat{B}\) has rows \({(1, 1)}\), \({(1, 0)}\) and \({(0, 1)}\), and singular values \(\sqrt3\) and \(1\). Find its Frobenius, 2-, 1-, max- and nuclear norms, and check the first against the singular values.
Show solution
Frobenius: four entries equal 1 and two equal 0, so \({\norm{\mat{B}}_F = \sqrt4 = 2}\), and indeed \({\sigma_1^2 + \sigma_2^2 = 3 + 1 = 4}\). The 2-norm is \({\sigma_1 = \sqrt3 \approx 1.73}\). The column sums are 2 and 2, so \({\norm{\mat{B}}_1 = 2}\). The row sums are 2, 1 and 1, so \({\norm{\mat{B}}_\infty = 2}\). The nuclear norm is \({{\sqrt3 + 1} \approx 2.73}\).
2.★★ Prove that the max-norm of a matrix is its largest row sum, \({\norm{\mat{A}}_\infty} = {\max_i \sum_j \abs{a_{ij}}}\).
Show solution
Let \({r_i = \sum_j \abs{a_{ij}}}\) and \(r_{\max}\) the largest. For any \(\vect{x}\), entry \(i\) of \(\mat{A}\vect{x}\) satisfies
because each \({\abs{x_j} \le \norm{\vect{x}}_\infty}\). The largest entry is then at most \({r_{\max}\norm{\vect{x}}_\infty}\), so no stretch exceeds \(r_{\max}\). To reach it, take the best row \(i\) and the vector \(\vect{x}\) with \({x_j = 1}\) where \({a_{ij} \ge 0}\) and \({x_j = -1}\) where \({a_{ij} < 0}\). Then \({\norm{\vect{x}}_\infty = 1}\) and entry \(i\) of \(\mat{A}\vect{x}\) is \({\sum_j \abs{a_{ij}} = r_{\max}}\). For \(\mat{M}\) this vector is \({(1, 1)}\), the corner in the figure.
3.★★★ Show that for any induced norm, every eigenvalue satisfies \({\abs{\lambda} \le \norm{\mat{A}}}\). Deduce that if \({\norm{\mat{A}} < 1}\) in some induced norm, then \(\mat{A}^k\vect{x} \to \vect{0}\) for every \(\vect{x}\).
Show solution
Let \({\mat{A}\vect{v} = \lambda\vect{v}}\) with \(\vect{v} \ne \vect{0}\). Then \({\abs{\lambda}\,\norm{\vect{v}}} = {\norm{\mat{A}\vect{v}}} \le {\norm{\mat{A}}\,\norm{\vect{v}}}\), and dividing by \({\norm{\vect{v}} > 0}\) gives \({\abs{\lambda} \le \norm{\mat{A}}}\). (For a complex eigenvalue the same lines work with complex vectors, and the 1-, 2- and max-norms keep their formulas.) For the second part, the submultiplicative rule gives \({\norm{\mat{A}^k\vect{x}}} \le {\norm{\mat{A}}^k\norm{\vect{x}}}\), and \({\norm{\mat{A}}^k \to 0}\) when \({\norm{\mat{A}} < 1}\). Lesson M03.9.1 uses this with the max-norm to show that no eigenvalue of a Markov matrix exceeds 1 in size.
Summary card
The size of a matrix is how much it can stretch, or how long its entries are.
Induced.\({\norm{\mat{A}}} = {\max \norm{\mat{A}\vect{x}}/\norm{\vect{x}}}\), so \({\norm{\mat{A}\vect{x}}} \le {\norm{\mat{A}}\norm{\vect{x}}}\).
2-norm\({= \sigma_1}\); 1-norm = largest column sum; max-norm = largest row sum; nuclear\({= \sum\sigma_i}\).
Products.\({\norm{\mat{A}\mat{B}}} \le {\norm{\mat{A}}\,\norm{\mat{B}}}\), and \({\norm{\mat{A}}_2} \le {\norm{\mat{A}}_F} \le {\sqrt{r}\,\norm{\mat{A}}_2}\).
Sources and next steps
Sources. Each was opened and checked against this lesson on 15 November 2026.
Sheldon Axler, Linear Algebra Done Right, 4th edition (open access). Section 7F: the bound \({\norm{T v} \le s_1\norm{v}}\) (7.82), the definition of the norm of a linear map as the largest \({\norm{Tv}}\) over vectors of length at most 1 (7.86), its norm properties (7.87), and the formulas \({\norm{T} =}\) largest singular value and largest \({\norm{Tv}}\) over unit vectors (7.88). Exercise 12 of section 8D names the Frobenius norm and warns that it is not this norm.
Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, chapter 2. Section 2.5 lists the properties of a norm and gives the Frobenius norm (2.33) as the most common way to measure a matrix in deep learning; section 2.10 writes it with the trace (2.49).
NumPy reference, numpy.linalg.norm: the table of ord values for matrices ('fro', 'nuc', inf as the largest row sum, 1 as the largest column sum, 2 as the largest singular value) and the Frobenius norm as the default.
Gilbert Strang, Lecture 8: Norms of vectors and matrices, MIT OpenCourseWare 18.065 (Spring 2018): the unit balls of the vector norms and the matrix norm as the largest growth factor, \({\max \norm{\mat{A}\vect{x}}/\norm{\vect{x}}}\).
Next.M03.8.2 The condition number divides the top of the stretch curve by its bottom, and finds out how far a small error can travel.
![A worked-example trace. On the left the matrix M with entries 3, 0, 4, 5, with its column sums 7 and 5 written below and its row sums 3 and 9 written at the side. On the right six boxes: Frobenius, root of 9 plus 0 plus 16 plus 25, equals root 50, about 7.07; M transpose M equals [[25, 20], [20, 25]] with eigenvalues 45 and 5; 2-norm sigma 1 equals root 45 equals 3 root 5, about 6.71; nuclear, 3 root 5 plus root 5 equals 4 root 5, about 8.94; 1-norm, the columns give 7 and 5, so 7; max-norm, the rows give 3 and 9, so 9.](figures/M03.8.1-F4.svg)
