Skip to main content

A sum of rank-one pieces

Multiply out S = Q Lambda Q-transpose and a symmetric matrix falls apart into a weighted sum of projections onto perpendicular lines. Use the pieces to act on a vector, to build S inverse, S to a power and the square root of S, and to glimpse low-rank approximation.
M03.3.3L2 Intermediate25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: M03.3.2 The spectral theorem M02.3 Dot product, norms, angles and projections

What you will be able to do
  • Derive S = lambda_1 q_1 q_1-transpose + … + lambda_n q_n q_n-transpose by multiplying Q Lambda Q-transpose column by row.
  • Explain why each piece q q-transpose is the projection onto a line, and compute S x by projecting, scaling and adding.
  • Compute S inverse, a power of S and the square root of S by applying the function to the eigenvalues only.
  • Explain why dropping the pieces with small eigenvalues gives a good approximation of lower rank.

The question

The last lesson wrote a symmetric matrix as three factors. A data scientist often wants something else: the inverse of a covariance matrix for a distance, its square root to whiten data, a high power for a long chain of steps, or a cheap copy that keeps only what matters. Is there one view of a symmetric matrix that makes all of these easy?

From the last lesson

1. State the spectral theorem.

Show answer

Every real symmetric matrix is \(\mat{S} = \mat{Q}\mat{\Lambda}\mat{Q}\T\), with \(\mat{Q}\) orthogonal (orthonormal eigenvectors as columns) and \(\mat{\Lambda}\) the real diagonal matrix of eigenvalues.

2. What does a symmetric 2 by 2 matrix with eigenvalues 3 and 1 do to the unit circle?

Show answer

It makes an ellipse with half-widths 3 and 1 along the two eigenvectors.

Intuition

To apply \(\mat{S}\) to a vector, you do not need its entries at all. Drop the vector at right angles onto each eigen-line, stretch each shadow by its eigenvalue, and add the results.

Two panels with the two pale orange eigen-lines of S, the q1 line labelled lambda 1 equals 3 and the q2 line labelled lambda 2 equals 1. Left, titled 1. Project onto each eigen-line: a blue arrow x equals (2, 0), dashed drop lines from its tip meeting the two eigen-lines at right angles, and vermillion arrows to the feet (1, 1) and (1, minus 1). Right, titled 2. Scale by lambda and add: the shadow (1, 1) is extended to 3 times (1, 1) along the q1 line, the shadow 1 times (1, minus 1) stays, a dashed parallelogram completes them, and a thick vermillion arrow S x equals (4, 2) is their sum. A line underneath reads: S x equals 3 (q1 dot x) q1 plus 1 (q2 dot x) q2, one piece for each eigen-line.
M03.3.3-F1

\(\mat{S}\) applied to \(\vect{x} = {(2, 0)}\) without its entries: project onto the two eigen-lines, scale the shadows by 3 and 1, add. The answer is \({(4, 2)}\), which is twice the first column of \(\mat{S}\), as it must be.

Each eigen-line contributes one piece, and the pieces never interfere, because the lines are perpendicular. That splitting is what this lesson writes as a sum of matrices.

Build it up

Step 1. A shadow is a matrix. Lesson M02.3.4 found the projection of \(\vect{x}\) onto the line of a unit vector \(\vect{q}\): \({(\vect{q} \cdot \vect{x})\,\vect{q}}\). Write the dot product as \({\vect{q}\T\vect{x}}\) and move the number to the other side of the vector:

\[ (\vect{q} \cdot \vect{x})\,\vect{q} = \vect{q}\,(\vect{q}\T\vect{x}) = (\vect{q}\vect{q}\T)\,\vect{x}. \]

The column \(\vect{q}\) times the row \(\vect{q}\T\) is a square matrix, the outer product \(\vect{q}\vect{q}\T\). It is the projection onto the line of \(\vect{q}\).

Step 2. What the piece looks like. For \(\vect{q}_1 = {(1, 1)/\sqrt{2}}\), entry \(ij\) of \(\vect{q}_1\vect{q}_1\T\) is \({(q_1)_i (q_1)_j} = {\tfrac12}\), so \(\vect{q}_1\vect{q}_1\T = \tfrac12\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}\). For \(\vect{q}_2 = {(1, -1)/\sqrt{2}}\) it is \(\tfrac12\begin{bmatrix} 1 & -1 \\ -1 & 1 \end{bmatrix}\). Each has rank one: every column is a multiple of \(\vect{q}\).

Two panels, each with six blue points, a pale orange line and a short orange arrow along it. Left, titled q1 q1 transpose: drop onto the q1 line: green arrows carry each blue point at right angles onto the line through (1, 1), where a vermillion dot marks its foot. Right, titled q2 q2 transpose: drop onto the q2 line: the same six points are carried onto the line through (1, minus 1). Below, the two matrices: q1 q1 transpose equals one half times rows 1, 1 and 1, 1; q2 q2 transpose equals one half times rows 1, minus 1 and minus 1, 1.
M03.3.3-F2

The two pieces of \(\mat{S}\) as maps. Each one drops every point of the plane onto its own line.

Step 3. Three facts about the pieces. Write \(\mat{P}_i = \vect{q}_i\vect{q}_i\T\).

  • Doing it twice changes nothing: \({\mat{P}_i\mat{P}_i} = {\vect{q}_i(\vect{q}_i\T\vect{q}_i)\vect{q}_i\T} = \mat{P}_i\), because \({\vect{q}_i\T\vect{q}_i} = 1\).
  • Two different pieces kill each other: for \({i \neq j}\), \({\mat{P}_i\mat{P}_j} = {\vect{q}_i(\vect{q}_i\T\vect{q}_j)\vect{q}_j\T} = \mat{0}\), because \({\vect{q}_i\T\vect{q}_j} = 0\).
  • All of them together do nothing: \({\mat{P}_1 + \cdots + \mat{P}_n} = {\mat{Q}\mat{Q}\T} = \mat{I}\), which is the next section’s expansion with every \(\lambda_i = 1\).

The formal statement

NoteThe spectral decomposition as a sum

If \(\mat{S} = \mat{Q}\mat{\Lambda}\mat{Q}\T\) with orthonormal eigenvectors \(\vect{q}_i\), then

\[ \mat{S} = \lambda_1\vect{q}_1\vect{q}_1\T + \lambda_2\vect{q}_2\vect{q}_2\T + \cdots + \lambda_n\vect{q}_n\vect{q}_n\T , \]

and for every vector \(\vect{x}\), \(\mat{S}\vect{x} = {\sum_i \lambda_i\,(\vect{q}_i \cdot \vect{x})\,\vect{q}_i}\).

Derivation. A product of matrices can be expanded column by row: if \(\mat{B}\) has columns \(\vect{b}_k\) and \(\mat{C}\) has rows \(\vect{c}_k\T\), then \(\mat{B}\mat{C} = {\sum_k \vect{b}_k\vect{c}_k\T}\). Entry \(ij\) of the right-hand side is \({\sum_k (\vect{b}_k)_i(\vect{c}_k)_j} = {\sum_k b_{ik}c_{kj}}\), which is the row-times-column rule for entry \(ij\) of \(\mat{B}\mat{C}\). Apply it with \(\mat{B} = \mat{Q}\mat{\Lambda}\), whose column \(k\) is \(\lambda_k\vect{q}_k\) (each column of \(\mat{Q}\) scaled), and \(\mat{C} = \mat{Q}\T\), whose row \(k\) is \(\vect{q}_k\T\):

\[ \begin{aligned} \mat{Q}\mat{\Lambda}\mat{Q}\T &= \sum_k (\lambda_k\vect{q}_k)\,\vect{q}_k\T \\ &= \sum_k \lambda_k\,\vect{q}_k\vect{q}_k\T . \end{aligned} \]

Multiplying by \(\vect{x}\) and using Step 1 gives the second form. \(\blacksquare\)

Functions of a symmetric matrix. The three facts of Step 3 make powers easy. Square the sum: every cross term \({\lambda_i\lambda_j\mat{P}_i\mat{P}_j}\) with \({i \neq j}\) is zero, and each \({\mat{P}_i\mat{P}_i}\) is \(\mat{P}_i\), so

\[ \mat{S}^2 = \sum_i \lambda_i^2\,\mat{P}_i . \]

Repeating the step gives \({\mat{S}^k} = {\sum_i \lambda_i^k\,\mat{P}_i}\) for every whole number \(k\) (exercise 3). The same idea defines other functions.

  • Inverse. If no eigenvalue is 0, let \(\mat{T} = {\sum_i \lambda_i^{-1}\mat{P}_i}\). Then \({\mat{S}\mat{T}} = {\sum_i \lambda_i\lambda_i^{-1}\mat{P}_i} = {\sum_i \mat{P}_i} = \mat{I}\), so \(\mat{T} = {\mat{S}^{-1}}\).
  • Square root. If no eigenvalue is negative, let \({\mat{S}^{1/2}} = {\sum_i \sqrt{\lambda_i}\,\mat{P}_i}\). Its square is \({\sum_i \lambda_i\mat{P}_i} = \mat{S}\), it is symmetric, and its eigenvalues \({\sqrt{\lambda_i}}\) are not negative. It is the only matrix with all three properties (Axler 7.39).
Three square panels, each with the pale orange eigen-lines of S, a blue unit circle and a vermillion ellipse. Left, titled S inverse: one third and 1: a narrow ellipse, one third wide along q1 and 1 along q2. Middle, titled S to the one half: root 3 and 1: an ellipse root 3 along q1 and 1 along q2. Right, titled S: 3 and 1: an ellipse 3 along q1 and 1 along q2. Orange arrows on the axes are labelled with the half-widths. Underneath: the same eigen-lines every time; only the half-widths change, from the eigenvalues 3 and 1.
M03.3.3-F4

\({\mat{S}^{-1}}\), \({\mat{S}^{1/2}}\) and \(\mat{S}\) share the eigenvectors of \(\mat{S}\). Only the stretch on each line changes: \({\tfrac13}\), \({\sqrt{3}}\) and 3 along \(\vect{q}_1\), and \(1\) along \(\vect{q}_2\) in all three.

Keeping only the big pieces. If some eigenvalues are much larger than the rest, the pieces with small \(\abs{\lambda_i}\) change \(\mat{S}\) little, and dropping them leaves a matrix of lower rank that does almost the same job. For \(\mat{S}\), keeping only \({3\mat{P}_1}\) leaves an error of exactly \({1\,\mat{P}_2}\). Module M03.6 makes this precise for any matrix, through the singular value decomposition; for a covariance matrix it is the core of principal component analysis.

Worked example

Split \(\mat{S}\) and the 3 by 3 matrix \(\mat{S}_3\) into pieces, then find \({\mat{S}^{-1}}\), \(\mat{S}^5\) and \({\mat{S}^{1/2}}\).

The 2 by 2 case. With the pieces of Step 2,

\[ \begin{aligned} &3\mat{P}_1 + 1\mat{P}_2 \\ &\quad= \tfrac32\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix} + \tfrac12\begin{bmatrix} 1 & -1 \\ -1 & 1 \end{bmatrix} \\ &\quad= \begin{bmatrix} \tfrac32 + \tfrac12 & \tfrac32 - \tfrac12 \\ \tfrac32 - \tfrac12 & \tfrac32 + \tfrac12 \end{bmatrix} \\ &\quad= \mat{S}. \end{aligned} \]

On \(\vect{x} = {(2, 0)}\): \(\mat{P}_1\vect{x} = {(1, 1)}\) and \(\mat{P}_2\vect{x} = {(1, -1)}\), so \(\mat{S}\vect{x} = {3\,(1, 1) + (1, -1)} = {(4, 2)}\), as in the anchor figure.

The 3 by 3 case. \(\mat{S}_3 = \begin{bmatrix} 2 & 1 & 1 \\ 1 & 2 & 1 \\ 1 & 1 & 2 \end{bmatrix}\) has eigenvalue 4 with \(\vect{q}_1 = {(1, 1, 1)/\sqrt{3}}\), and the repeated eigenvalue 1 on the plane \({x + y + z} = 0\) (last lesson). Every entry of \(\vect{q}_1\vect{q}_1\T\) is \({\tfrac{1}{\sqrt3} \cdot \tfrac{1}{\sqrt3}} = {\tfrac13}\). The two pieces for the eigenvalue 1 add up to \({\mat{P}_2 + \mat{P}_3} = {\mat{I} - \mat{P}_1}\), by the third fact of Step 3, whichever orthonormal pair is chosen in the plane.

A matrix equation. S3, with rows 2, 1, 1 and 1, 2, 1 and 1, 1, 2, equals 4 times one third times the matrix of all ones, labelled q1 q1 transpose: onto the line of (1, 1, 1), plus 1 times one third times the matrix with rows 2, minus 1, minus 1 and minus 1, 2, minus 1 and minus 1, minus 1, 2, labelled q2 q2 transpose plus q3 q3 transpose: onto the plane x plus y plus z equals 0. A yellow box underneath: diagonal: four thirds plus two thirds equals 2; off the diagonal: four thirds minus one third equals 1.
M03.3.3-F3

\(\mat{S}_3\) as 4 times the projection onto a line plus 1 times the projection onto the plane perpendicular to it. The yellow box checks every entry.

Functions of \(\mat{S}\). Apply each function to 3 and 1 and keep the pieces:

\[ \begin{aligned} \mat{S}^{-1} &= \tfrac13\mat{P}_1 + \mat{P}_2 \\ &= \begin{bmatrix} \tfrac16 + \tfrac12 & \tfrac16 - \tfrac12 \\ \tfrac16 - \tfrac12 & \tfrac16 + \tfrac12 \end{bmatrix} \\ &= \tfrac13\begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix}, \\[4pt] \mat{S}^5 &= 243\,\mat{P}_1 + \mat{P}_2 \\ &= \begin{bmatrix} 122 & 121 \\ 121 & 122 \end{bmatrix}, \\[4pt] \mat{S}^{1/2} &= \sqrt3\,\mat{P}_1 + \mat{P}_2 \\ &= \tfrac12\begin{bmatrix} \sqrt3 + 1 & \sqrt3 - 1 \\ \sqrt3 - 1 & \sqrt3 + 1 \end{bmatrix}. \end{aligned} \]

Checks. \(\mat{S}\) times the first column of \({\mat{S}^{-1}}\) is \({\tfrac13(4 - 1,\; 2 - 2)} = {(1, 0)}\). The entries of \(\mat{S}^5\) are \({243/2 + 1/2} = 122\) and \({243/2 - 1/2} = 121\), as lesson M02.9.4 found the long way. And the top-left entry of the square of \({\mat{S}^{1/2}}\) is \({\tfrac14\big((\sqrt3 + 1)^2 + (\sqrt3 - 1)^2\big)} = {\tfrac14(4 + 2\sqrt3 + 4 - 2\sqrt3)} = 2\), while the top-right is \({\tfrac14 \cdot 2(\sqrt3 + 1)(\sqrt3 - 1)} = {\tfrac14 \cdot 2 \cdot 2} = 1\). Numerically, \(\mat{S}^{1/2} \approx \begin{bmatrix} 1.366 & 0.366 \\ 0.366 & 1.366 \end{bmatrix}\).

Code

From scratch: rebuild a symmetric matrix from its pieces, then apply any function through the eigenvalues.

import math
import numpy as np

def pieces(S):
    """Eigenvalues and the projection matrices q q^T of a symmetric matrix."""
    lam, Q = np.linalg.eigh(S)
    return lam, [np.outer(Q[:, i], Q[:, i]) for i in range(len(lam))]

def apply_function(S, f):
    """f(S) = sum of f(lambda_i) q_i q_i^T."""
    lam, P = pieces(S)
    return sum(f(l) * p for l, p in zip(lam, P))

S = np.array([[2.0, 1.0], [1.0, 2.0]])
S3 = np.array([[2.0, 1.0, 1.0], [1.0, 2.0, 1.0], [1.0, 1.0, 2.0]])
lam, P = pieces(S)
print("eigenvalues:", lam.round(10))
print("pieces:\n", P[0].round(6), "\n", P[1].round(6))
print("sum of lambda_i P_i:\n", sum(l * p for l, p in zip(lam, P)).round(10))
print("S x by pieces:", sum(l * (p @ [2.0, 0.0]) for l, p in zip(lam, P)).round(10))
eigenvalues: [1. 3.]
pieces:
 [[ 0.5 -0.5]
 [-0.5  0.5]] 
 [[0.5 0.5]
 [0.5 0.5]]
sum of lambda_i P_i:
 [[2. 1.]
 [1. 2.]]
S x by pieces: [4. 2.]

The order of eigh is ascending, so its first piece belongs to the eigenvalue 1. Now the three functions, each checked against an independent library route.

from scipy.linalg import sqrtm

S_inv = apply_function(S, lambda l: 1 / l)
S_5 = apply_function(S, lambda l: l ** 5)
S_half = apply_function(S, math.sqrt)
print("S^-1 =\n", S_inv.round(6), "\n np.linalg.inv agrees:", np.allclose(S_inv, np.linalg.inv(S)))
print("S^5 =\n", S_5.round(6), "\n matrix_power agrees:", np.allclose(S_5, np.linalg.matrix_power(S, 5)))
print("S^(1/2) =\n", S_half.round(4), "\n sqrtm agrees:", np.allclose(S_half, sqrtm(S)))
print("(S^(1/2))^2 =\n", (S_half @ S_half).round(10))
S^-1 =
 [[ 0.666667 -0.333333]
 [-0.333333  0.666667]] 
 np.linalg.inv agrees: True
S^5 =
 [[122. 121.]
 [121. 122.]] 
 matrix_power agrees: True
S^(1/2) =
 [[1.366 0.366]
 [0.366 1.366]] 
 sqrtm agrees: True
(S^(1/2))^2 =
 [[2. 1.]
 [1. 2.]]

Last, the glimpse of low rank: keep only the biggest piece of a symmetric matrix and measure what is lost. The size of a matrix here is the square root of the sum of its squared entries, a measure that module M03.8 studies.

rng = np.random.default_rng(1)
X = rng.normal(size=(200, 3)) @ np.array([[3.0, 0.0, 0.0], [1.0, 1.0, 0.0], [0.5, 0.2, 0.3]])
C = np.cov(X, rowvar=False)                        # a 3 by 3 covariance matrix
lam, P = pieces(C)
keep_one = lam[-1] * P[-1]                         # the piece with the largest eigenvalue
size = lambda M: math.sqrt((M ** 2).sum())
print("eigenvalues of C:", lam.round(3))
print("share of the size lost by keeping one piece:", round(size(C - keep_one) / size(C), 3))
eigenvalues of C: [0.077 0.809 8.837]
share of the size lost by keeping one piece: 0.092

One piece out of three keeps almost all of this covariance matrix. With the eigenvalues numbered from the largest, \({\lambda_1 \ge \lambda_2 \ge \lambda_3}\), the error is the size of the two dropped pieces, which is \({\sqrt{\lambda_2^2 + \lambda_3^2}}\), because the pieces are perpendicular to one another.

Explore

The spectral ellipse in lesson M03.3.2 has a Pieces switch. Turn it on and turn the angle slider: the chosen vector is projected onto each eigen-line, each shadow is stretched by its eigenvalue, and the two stretched shadows add up to the image. Make one eigenvalue negative and watch its piece point backwards.

Pitfalls

Two square panels, each with a blue unit circle, a pale orange line along q1 and a dashed black ellipse, the image of the circle under S. Left, marked with a cross and titled Root of each entry: a long, very thin vermillion ellipse along q1 that reaches far beyond the dashed ellipse, labelled too long, too thin, and the note squared, it misses the dashed target S. Right, marked with a tick and titled Root of each eigenvalue: the vermillion ellipse lies exactly on the dashed one, with the note squared, it lands exactly on S. Below: entrywise root, squared, is about rows 3, 2.83 and 2.83, 3; S to the one half, squared, equals rows 2, 1 and 1, 2. Underneath: blue, the unit circle; dashed, what S makes of it; vermillion, what the square of the root makes of it.
M03.3.3-F5

Wrong beside right. Taking the square root of each entry of \(\mat{S}\) gives a matrix whose square is not \(\mat{S}\). Taking the square root of each eigenvalue does.

“np.sqrt(S) is the square root of the matrix, and S ** 2 is its square.”

Both work entry by entry. np.sqrt(S) has entries \({\sqrt2}\) and 1, and its square is about \(\begin{bmatrix} 3 & 2.83 \\ 2.83 & 3 \end{bmatrix}\), not \(\mat{S}\). S ** 2 has entries 4 and 1, while \({\mat{S}\mat{S}} = \begin{bmatrix} 5 & 4 \\ 4 & 5 \end{bmatrix}\). Use S @ S, np.linalg.matrix_power, scipy.linalg.sqrtm, or the pieces.

“Any eigenvector will do in the piece \({\lambda\,\vect{v}\vect{v}\T}\).”

Only a unit vector gives a projection. With \(\vect{v} = {(1, 1)}\), \({\vect{v}\vect{v}\T} = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}\) is twice \(\mat{P}_1\), and \({3\vect{v}\vect{v}\T + \ldots}\) overshoots. Divide by \({\vect{v}\T\vect{v}}\), or normalise first.

“Every symmetric matrix has a square root.”

Not a real symmetric one. The square of a real symmetric matrix \(\mat{R}\) has the eigenvalues \({\mu_i^2 \ge 0}\), where the \(\mu_i\) are the real eigenvalues of \(\mat{R}\). The running saddle has the eigenvalue \({-1}\), so no real symmetric matrix squares to it. The square root of this lesson needs every eigenvalue to be at least 0: the positive semidefinite matrices of module M03.4, which include every covariance matrix.

Check yourself

Recall

1. Write a symmetric matrix as a sum of pieces, and say what each piece is.

Show answer

\(\mat{S} = {\sum_i \lambda_i\vect{q}_i\vect{q}_i\T}\). Each \(\vect{q}_i\vect{q}_i\T\) is the projection onto the line of the unit eigenvector \(\vect{q}_i\), scaled by its eigenvalue.

2. Why is \(\vect{q}\vect{q}\T\vect{x}\) the projection of \(\vect{x}\) onto the line of a unit vector \(\vect{q}\)?

Show answer

\({\vect{q}\vect{q}\T\vect{x}} = {\vect{q}(\vect{q}\T\vect{x})} = {(\vect{q} \cdot \vect{x})\vect{q}}\), the formula for the projection onto a unit vector.

3. What are \(\mat{P}_1\mat{P}_1\), \(\mat{P}_1\mat{P}_2\) and \({\mat{P}_1 + \mat{P}_2}\) for the pieces of a symmetric 2 by 2 matrix?

Show answer

\(\mat{P}_1\), the zero matrix, and \(\mat{I}\).

4. How do you compute \({\mat{S}^{-1}}\) from the pieces, and when does it exist?

Show answer

\({\mat{S}^{-1}} = {\sum_i \lambda_i^{-1}\mat{P}_i}\), which exists when no eigenvalue is 0.

5. Which matrix is \({\mat{S}^{1/2}}\) for the running \(\mat{S}\), roughly?

Show answer

\(\sqrt3\,\mat{P}_1 + \mat{P}_2 \approx \begin{bmatrix} 1.366 & 0.366 \\ 0.366 & 1.366 \end{bmatrix}\).

Exercises

1. ★ Write the running saddle \(\begin{bmatrix} 1 & 2 \\ 2 & 1 \end{bmatrix}\) as a sum of two pieces, and use them to compute its effect on \({(2, 0)}\).

Show solution

Its eigenvalues are 3 and \({-1}\) with the same eigenvectors as \(\mat{S}\), so the saddle is \({3\mat{P}_1 - \mat{P}_2}\). Check: \({\tfrac32\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix} - \tfrac12\begin{bmatrix} 1 & -1 \\ -1 & 1 \end{bmatrix}}\) has diagonal entries \({\tfrac32 - \tfrac12} = 1\) and off-diagonal entries \({\tfrac32 + \tfrac12} = 2\). On \({(2, 0)}\): \({3\,(1, 1) - (1, -1)} = {(2, 4)}\), which is the saddle’s first column times 2.

2. ★★ Find the symmetric square root, with no negative eigenvalues, of \(\mat{M} = \begin{bmatrix} 5 & 4 \\ 4 & 5 \end{bmatrix}\).

Show solution

\(\mat{M}\) sends \({(1, 1)}\) to \({(9, 9)}\) and \({(1, -1)}\) to \({(1, -1)}\), so it has the eigenvalues 9 and 1 with the pieces \(\mat{P}_1\) and \(\mat{P}_2\) of \(\mat{S}\). Its square root is \({\sqrt9\,\mat{P}_1 + \sqrt1\,\mat{P}_2} = {3\mat{P}_1 + \mat{P}_2} = \mat{S}\). Indeed \({\mat{S}\mat{S}} = \begin{bmatrix} 4 + 1 & 2 + 2 \\ 2 + 2 & 1 + 4 \end{bmatrix} = \mat{M}\).

3. ★★★ Prove that \({\mat{S}^k} = {\sum_i \lambda_i^k\mat{P}_i}\) for every whole number \({k \ge 1}\), using only the three facts of Step 3.

Show solution

By induction on \(k\). For \(k = 1\) it is the spectral decomposition. Suppose it holds for \(k\). Then

\[ \begin{aligned} \mat{S}^{k+1} &= \mat{S}^k\mat{S} \\ &= \Big(\sum_i \lambda_i^k\mat{P}_i\Big)\Big(\sum_j \lambda_j\mat{P}_j\Big) \\ &= \sum_i\sum_j \lambda_i^k\lambda_j\,\mat{P}_i\mat{P}_j . \end{aligned} \]

Every term with \({i \neq j}\) is zero, because \({\mat{P}_i\mat{P}_j} = \mat{0}\). The terms with \(i = j\) are \({\lambda_i^{k+1}\mat{P}_i\mat{P}_i} = {\lambda_i^{k+1}\mat{P}_i}\). So \({\mat{S}^{k+1}} = {\sum_i \lambda_i^{k+1}\mat{P}_i}\), which is the claim for \({k + 1}\).

Summary card

A symmetric matrix is a weighted sum of projections onto perpendicular lines.

The anchor figure again: x equals (2, 0) is projected onto the two eigen-lines of S, the shadows are scaled by 3 and 1, and they add up to S x equals (4, 2).
M03.3.3-F1

Project, scale, add.

  • Pieces. \(\mat{S} = {\sum_i \lambda_i\vect{q}_i\vect{q}_i\T}\), with \({\vect{q}\vect{q}\T\vect{x}} = {(\vect{q} \cdot \vect{x})\vect{q}}\).
  • Rules. \({\mat{P}_i^2} = \mat{P}_i\), \({\mat{P}_i\mat{P}_j} = \mat{0}\), \({\sum_i \mat{P}_i} = \mat{I}\).
  • Functions. \({f(\mat{S})} = {\sum_i f(\lambda_i)\,\mat{P}_i}\): inverse, powers, square root.
  • Running example. \(\mat{S} = {3\mat{P}_1 + \mat{P}_2}\), so \({\mat{S}^{1/2}} = {\sqrt3\,\mat{P}_1 + \mat{P}_2}\).
  • Low rank. Drop the pieces with small \(\abs{\lambda_i}\) (module M03.6).

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • Gilbert Strang, Symmetric matrices and positive definiteness, MIT OpenCourseWare 18.06SC, lecture 25 summary, part “Projection onto eigenvectors”: writing \({\mat{Q}\mat{\Lambda}\mat{Q}\T}\) column by row gives \({\lambda_1\vect{q}_1\vect{q}_1\T + \cdots + \lambda_n\vect{q}_n\vect{q}_n\T}\), and each \({\vect{q}_k\vect{q}_k\T}\) is a projection matrix, so every symmetric matrix is a combination of perpendicular projection matrices.
  • Sheldon Axler, Linear Algebra Done Right, 4th edition (open access). Result 7.39: every positive operator has exactly one positive square root; notation 7.40 writes it \({\sqrt{T}}\). (For a self-adjoint operator, Axler’s “positive” means that no eigenvalue is negative.)
  • Marc Peter Deisenroth, A. Aldo Faisal and Cheng Soon Ong, Mathematics for Machine Learning (free PDF), Section 4.4, eigendecomposition and diagonalisation: a symmetric matrix is \({\mat{P}\mat{D}\mat{P}\T}\) with orthonormal eigenvectors, and powers are taken through the diagonal factor.
  • SciPy reference, scipy.linalg.sqrtm: the matrix square root, used here only as an independent check.

Next. M03.3.4 The Rayleigh quotient and the extreme directions asks which unit vector a symmetric matrix stretches most, and answers with the largest eigenvalue: the first step of principal component analysis.

This lesson unlocks

M03.3.4 The Rayleigh quotient and the extreme directions M03.6 Low-rank approximation and PCA

Your progress is saved in this browser only. No account, no tracking.
Back to top