Skip to main content

Index notation and the summation convention

Write any product of matrices and vectors as letters with indices, drop the summation sign, and prove identities by moving letters: transposes of products, traces, quadratic forms and the gradient of a quadratic form.
M03.10.2L2 Intermediate25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: M03.10.1 Tensors as arrays with many axes M04.10 Partial derivatives, a first look

What you will be able to do
  • Write matrix products, transposes, traces, dot and outer products in index notation, and name the free and the dummy indices.
  • Check an index expression for validity: free indices match on both sides and no index appears more than twice in a term.
  • Use the Kronecker delta to substitute one index for another.
  • Derive (AB)-transpose = B-transpose A-transpose, tr(AB) = tr(BA) and the gradient of x-transpose S x in index notation.

The question

A rule quoted everywhere says the gradient of \(\vect{x}\T \mat{S} \vect{x}\) is \(2\mat{S}\vect{x}\). For the running matrix \(\mat{A}\), rows \((4, 1)\) and \((2, 3)\), at \(\vect{x} = (1, 2)\), that formula gives \((12, 16)\), but the true gradient is \((14, 15)\). Which step of the rule silently assumed something, and is there a way of writing sums in which such slips cannot hide?

From the last lesson

1. An array has shape (B, C, H, W) = (32, 3, 28, 28). What is its order, and how many numbers does it hold?

Show answer

Order 4. It holds \({32 \cdot 3 \cdot 28 \cdot 28} = 75{,}264\) numbers.

2. Which call turns a channels-first image of shape (3, 2, 2) into channels-last, and why is reshape(2, 2, 3) wrong?

Show answer

x.transpose(1, 2, 0), with \(y[h, w, c] = x[c, h, w]\). The reshape keeps the reading order, so its first pixel holds the first three red values, 1, 2 and 3.

Intuition

Every entry of a matrix product is a sum: \(c_{ik} = \sum_j a_{ij} b_{jk}\). Look at which letters appear where. The letter \(j\) appears in both factors and not in the answer: it is used up by the sum. The letters \(i\) and \(k\) appear once each, and they say which entry of the answer we are computing.

A blue 2 by 3 matrix A with rows 1 2 3 and 0 1 4, its first row shaded yellow. Above and to the right, a sky-blue 3 by 2 matrix B with rows 1 0, 2 1 and 0 2, its second column shaded yellow. Below B and right of A, a vermillion 2 by 2 matrix C equals AB with rows 5 8 and 2 9, the entry 8 in row 1, column 2 shaded yellow. Measuring lines mark the columns of A and the rows of B with the letter j, the rows of A and C with i, and the columns of B with k. Beside it: c sub i k equals a sub i j times b sub j k; i and k appear once, free, they name the entry of the result; j appears twice, summed, it runs along the row and the column; c sub 1 2 equals 1 times 0 plus 2 times 1 plus 3 times 2 equals 8.
M03.10.2-F1

The rule of index notation in one picture. The repeated letter \(j\) runs along a row of \(\mat{A}\) and down a column of \(\mat{B}\) and is summed away; the two letters that appear once, \(i\) and \(k\), are the address of the answer.

Since the repeated letter always signals a sum, we can stop writing the \(\sum\). That is the whole idea of the summation convention, which Einstein introduced in his 1916 paper on general relativity: \(c_{ik} = a_{ij} b_{jk}\) means \(\sum_j a_{ij} b_{jk}\). Each axis of each array gets a letter, and the letters alone say what is multiplied and what is added.

Build it up

Step 1. A matrix times a vector, with the sum written. For \(\vect{y} = \mat{A}\vect{x}\) each entry is \({y_i = \sum_j a_{ij} x_j}\). The letter \(i\) picks a row; the letter \(j\) runs along that row and along \(\vect{x}\).

Two panels, titled free index i equals 1 and free index i equals 2. In each, three rows of three cells under column headings j equals 1, 2, 3: a blue row of A (1 2 3 in the first panel, 0 1 4 in the second), a sky-blue row x sub j holding 2 1 1, and a vermillion row of products (2 2 3, then 0 1 4). A green arrow leads from the products to a yellow cell: y sub 1 equals 7 and y sub 2 equals 5. Below: sum over j, 2 plus 2 plus 3 equals 7, and 0 plus 1 plus 4 equals 5.
M03.10.2-F2

\(y_i = a_{ij} x_j\) for the matrix of the anchor figure and \(\vect{x} = (2, 1, 1)\). The free index \(i\) chooses the row; the dummy index \(j\) runs along it and is summed.

Step 2. Drop the sum sign. Write \({y_i = a_{ij} x_j}\). Rule: an index that appears twice in a term is summed over all its values.

Step 3. Free and dummy. An index that appears once in a term is free: the equation holds for each of its values, so it names an entry of the result. An index that appears twice is a dummy: it is summed and is gone from the result. In \({y_i = a_{ij} x_j}\), \(i\) is free and \(j\) is a dummy.

Step 4. Free indices must match. Both sides of an equation, and every term of a sum, carry the same free indices. \({y_i = a_{ij} x_j}\) is fine: \(i\) on both sides. \({y_i = a_{jk} x_k}\) is nonsense: the right side has free index \(j\), the left side \(i\).

Step 5. Dummies can be renamed. \({a_{ij} x_j}\) and \({a_{ik} x_k}\) are the same sum, just as \(\sum_j\) and \(\sum_k\) give the same total. A dummy letter is a counter and nothing more.

Step 6. Never three times. In \({a_{ij} x_j y_j}\) the letter \(j\) appears three times, and the convention cannot say what is meant. Write such a sum with an explicit \(\sum_j\).

The formal statement

NoteDefinitions: summation convention, free and dummy indices, Kronecker delta
  • Summation convention. In a product of indexed quantities, an index that appears exactly twice is summed over its whole range. No index may appear more than twice in one term.
  • A free index appears once in each term; both sides of an equation carry the same free indices. A dummy index appears twice in a term and may be renamed to any unused letter.
  • The Kronecker delta is \(\delta_{ij} = 1\) when \(i = j\) and \(0\) otherwise: the entries of the identity matrix.

Four results follow, each in a few lines of letters.

Result 1: the delta renames an index. In \({a_{ij} \delta_{jk}}\), summed over \(j\), every term with \(j \neq k\) is zero, and the term \(j = k\) is \(a_{ik} \cdot 1\). So

\[ a_{ij}\, \delta_{jk} = a_{ik} . \]

In matrix language this says \(\mat{A}\mat{I} = \mat{A}\). Also \(\delta_{ii} = 1 + \dots + 1 = n\), the trace of the \(n\) by \(n\) identity.

Left: the 3 by 3 identity matrix drawn as a grid, titled delta sub j k, the identity matrix, with the diagonal ones shaded green and the rows labelled j equals 1 to 3 and columns k equals 1 to 3; captions read 1 where j equals k, 0 elsewhere, and delta sub j j equals 1 plus 1 plus 1 equals 3. Right, titled a sub 1 j times delta sub j 2, summed over j: three lines a 1 1 delta 1 2 equals 1 times 0, a 1 2 delta 2 2 equals 2 times 1 and a 1 3 delta 3 2 equals 3 times 0, with result cells 0, 2 (yellow) and 0. Below: equals 2 equals a sub 1 2; only j equals 2 survives.
M03.10.2-F5

The Kronecker delta at work on the first row \((1, 2, 3)\) of the anchor’s matrix: in \({a_{1j} \delta_{j2}}\) only the term \(j = 2\) survives, so the sum is \(a_{12} = 2\).

Result 2: the transpose of a product. The transpose swaps the indices, \((\mat{A}\T)_{ij} = a_{ji}\). Let \(\mat{C} = \mat{A}\mat{B}\), so \({c_{ik} = a_{ij} b_{jk}}\). Then

\[ \begin{aligned} (\mat{C}\T)_{ki} &= c_{ik} = a_{ij}\, b_{jk} \\ &= b_{jk}\, a_{ij} \\ &= (\mat{B}\T)_{kj}\, (\mat{A}\T)_{ji} \\ &= (\mat{B}\T \mat{A}\T)_{ki} . \end{aligned} \]

The second line swaps two numbers, which always commute. The third reads each factor through the transpose. The last recognises a matrix product: the dummy \(j\) sits last on the left factor and first on the right one. So \((\mat{A}\mat{B})\T = \mat{B}\T \mat{A}\T\).

Result 3: the trace of a product. The trace sums the diagonal, \(\operatorname{tr}(\mat{A}) = a_{ii}\). Then

\[ \begin{aligned} \operatorname{tr}(\mat{A}\mat{B}) &= (\mat{A}\mat{B})_{ii} = a_{ij}\, b_{ji} \\ &= b_{ji}\, a_{ij} = (\mat{B}\mat{A})_{jj} \\ &= \operatorname{tr}(\mat{B}\mat{A}) . \end{aligned} \]

Both traces add up the same products \({a_{ij} b_{ji}}\) over all \(i\) and \(j\).

Result 4: quadratic forms, outer products and a gradient. A quadratic form has two dummies, \(\vect{x}\T \mat{S} \vect{x} = x_i\, s_{ij}\, x_j\): a single number. An outer product has none, \((\vect{x}\vect{y}\T)_{ij} = x_i y_j\): two free indices, so an order-2 result from two order-1 inputs. To differentiate, use \(\pd{x_i}{x_k} = \delta_{ik}\) (each coordinate moves only with itself) and the product rule:

\[ \begin{aligned} \pd{}{x_k}\big(x_i\, s_{ij}\, x_j\big) &= \delta_{ik}\, s_{ij}\, x_j \\ &\quad + x_i\, s_{ij}\, \delta_{jk} \\ &= s_{kj}\, x_j + x_i\, s_{ik} \\ &= s_{kj}\, x_j + s_{jk}\, x_j \\ &= (s_{kj} + s_{jk})\, x_j . \end{aligned} \]

The first two lines are the product rule; the third uses Result 1 twice. The fourth renames the dummy \(i\) as \(j\). In matrix form the gradient is \({(\mat{S} + \mat{S}\T)\vect{x}}\), and only when \(\mat{S}\) is symmetric does it become \(2\mat{S}\vect{x}\). That answers the opening question: “\(2\mat{S}\vect{x}\)” assumed \(s_{jk} = s_{kj}\). The full matrix calculus is the subject of M05.7.

Go deeper: why the delta is the derivative of a coordinate

The function \(x \mapsto x_i\) reads off coordinate \(i\). Moving \(x_k\) by a small amount \(h\) while the other coordinates stay put changes \(x_i\) by \(h\) when \(i = k\) and by \(0\) otherwise. The rate of change is therefore \(1\) or \(0\), which is \(\delta_{ik}\). This is the partial derivative of M04.10.2, one coordinate at a time.

Worked example

Use the running matrices \(\mat{A}\), rows \((4, 1)\) and \((2, 3)\), and \(\mat{S}\), rows \((2, 1)\) and \((1, 2)\).

One entry of a product. With \({c_{ik} = a_{ij} s_{jk}}\), the entry \(c_{12}\) sums over \(j\):

\[ \begin{aligned} c_{12} &= a_{11}\, s_{12} + a_{12}\, s_{22} \\ &= 4 \cdot 1 + 1 \cdot 2 = 6 . \end{aligned} \]

The full products are \(\mat{A}\mat{S}\), rows \((9, 6)\) and \((7, 8)\), and \(\mat{S}\mat{A}\), rows \((10, 5)\) and \((8, 7)\). They differ.

The traces agree. By Result 3, both traces are the sum of \({a_{ij} s_{ji}}\) over the four pairs \((i, j)\):

\[ \begin{aligned} a_{ij}\, s_{ji} &= 4 \cdot 2 + 1 \cdot 1 + 2 \cdot 1 + 3 \cdot 2 \\ &= 8 + 1 + 2 + 6 = 17 . \end{aligned} \]

Three blocks. Left, the vermillion product AS with rows 9 6 and 7 8, its diagonal 9 and 8 shaded yellow, captioned trace 9 plus 8 equals 17. Right, the vermillion product SA with rows 10 5 and 8 7, diagonal 10 and 7 shaded yellow, captioned trace 10 plus 7 equals 17. In the middle, a grid titled terms a sub i j times s sub j i with rows i equals 1 and 2 holding 4 times 2, 1 times 1 and 2 times 1, 3 times 2, captioned 8 plus 1 plus 2 plus 6 equals 17. Green arrows point from both products to the middle grid. Below: AS is not SA, yet the diagonal entries of both are made of the same products; rows of the grid add to the diagonal of AS, 9 and 8, columns to that of SA, 10 and 7.
M03.10.2-F3

\(\operatorname{tr}(\mat{A}\mat{S}) = \operatorname{tr}(\mat{S}\mat{A}) = 17\), term by term. The rows of the grid of products add to the diagonal of \(\mat{A}\mat{S}\); its columns add to the diagonal of \(\mat{S}\mat{A}\).

Check: \(\operatorname{tr}(\mat{A}\mat{S}) = 9 + 8 = 17\) and \(\operatorname{tr}(\mat{S}\mat{A}) = 10 + 7 = 17\).

The gradient of the opening question. Let \(f(\vect{x}) = \vect{x}\T \mat{A} \vect{x}\). Written out, \(f = 4x_1^2 + 3x_1 x_2 + 3x_2^2\), because the two cross terms \(1 \cdot x_1 x_2\) and \({2 \cdot x_2 x_1}\) add up. Result 4 gives \({(\mat{A} + \mat{A}\T)\vect{x}}\) with \(\mat{A} + \mat{A}\T\) having rows \((8, 3)\) and \((3, 6)\), so at \(\vect{x} = (1, 2)\)

\[ \begin{aligned} \nabla f &= (8 \cdot 1 + 3 \cdot 2,\; 3 \cdot 1 + 6 \cdot 2) \\ &= (14, 15) . \end{aligned} \]

Directly: \(\pd{f}{x_1} = 8x_1 + 3x_2 = 14\) and \(\pd{f}{x_2} = 3x_1 + 6x_2 = 15\). The formula \(2\mat{A}\vect{x} = (12, 16)\) is wrong because \(\mat{A}\) is not symmetric.

Code

Each identity, checked with plain loops over the indices, so that every sum is visible.

import numpy as np

def matmul_loops(A, B):
    """c_ik = a_ij b_jk: loop over the free indices i, k and sum over the dummy j."""
    n, m = A.shape
    m2, p = B.shape
    assert m == m2, "the dummy index must have one range"
    C = np.zeros((n, p))
    for i in range(n):
        for k in range(p):
            for j in range(m):
                C[i, k] += A[i, j] * B[j, k]
    return C

def trace_loops(A):
    return sum(A[i, i] for i in range(A.shape[0]))     # a_ii

A = np.array([[4.0, 1.0], [2.0, 3.0]])
S = np.array([[2.0, 1.0], [1.0, 2.0]])
AS, SA = matmul_loops(A, S), matmul_loops(S, A)
print("AS =", AS.tolist(), "  SA =", SA.tolist())
print("tr(AS) =", trace_loops(AS), "  tr(SA) =", trace_loops(SA))
print("sum of a_ij s_ji:", sum(A[i, j] * S[j, i] for i in range(2) for j in range(2)))
AS = [[9.0, 6.0], [7.0, 8.0]]   SA = [[10.0, 5.0], [8.0, 7.0]]
tr(AS) = 17.0   tr(SA) = 17.0
sum of a_ij s_ji: 17.0

Now the transpose rule on rectangular matrices, the quadratic form and its gradient, compared with a numerical derivative.

rng = np.random.default_rng(0)
P, Q = rng.normal(size=(2, 3)), rng.normal(size=(3, 4))
print("(PQ)^T = Q^T P^T:", np.allclose(matmul_loops(P, Q).T, matmul_loops(Q.T, P.T)))

def quad(M, x):
    """x_i m_ij x_j, both indices dummies."""
    return sum(x[i] * M[i, j] * x[j] for i in range(len(x)) for j in range(len(x)))

def grad_index(M, x):
    """(m_kj + m_jk) x_j for each free k."""
    n = len(x)
    return np.array([sum((M[k, j] + M[j, k]) * x[j] for j in range(n)) for k in range(n)])

x = np.array([1.0, 2.0])
h = 1e-6
numeric = np.array([(quad(A, x + h * e) - quad(A, x - h * e)) / (2 * h) for e in np.eye(2)])
print("f(x) =", quad(A, x), "  index formula:", grad_index(A, x), "  numerical:", numeric.round(6))
print("2Ax =", 2 * A @ x, "  outer product x y^T:", np.multiply.outer(x, [3.0, 4.0]).tolist())
(PQ)^T = Q^T P^T: True
f(x) = 22.0   index formula: [14. 15.]   numerical: [14. 15.]
2Ax = [12. 16.]   outer product x y^T: [[3.0, 4.0], [6.0, 8.0]]

The numerical derivative agrees with \((14, 15)\) to six decimals, and the last check confirms that for the symmetric \(\mat{S}\) the index formula reduces to \(2\mat{S}\vect{x}\). The next lesson turns these loops into one general function, einsum.

Explore

The einsum explorer of M03.10.3 draws every expression of this lesson as a subscript string. Try ij,jk->ik (Result 2’s product), ii-> (the trace) and ij->ji (the transpose), and watch which letters it marks as summed.

Pitfalls

Two square panels, each showing the level curves of f equals x squared plus 4 x y plus y squared, a saddle whose curves are hyperbolas, with the curve f equals 1 drawn thicker and its tangent at the point (1, 0) dashed. Left, marked with a cross and titled 2 M x, not across the curve: a vermillion arrow 2 M x equals (2, 0) leaves (1, 0) along the horizontal axis, only about 27 degrees off the dashed tangent. Right, marked with a tick and titled M plus M transpose times x, at a right angle: a green arrow (M plus M transpose) x equals (2, 4) leaves (1, 0) at a right angle to the tangent, shown by a small square mark.
M03.10.2-F4

Wrong beside right, for \(\mat{M}\) with rows \((1, 4)\) and \((0, 1)\), whose quadratic form is the running saddle \(x^2 + 4xy + y^2\). The true gradient at \((1, 0)\) is \((2, 4)\), at right angles to the level curve; \(2\mat{M}\vect{x} = (2, 0)\) is only about 27° off the tangent.

“The gradient of \(\vect{x}\T \mat{M} \vect{x}\) is \(2\mat{M}\vect{x}\).”

Only for symmetric \(\mat{M}\). In general it is \({(\mat{M} + \mat{M}\T)\vect{x}}\) (Result 4). A quadratic form only sees the symmetric part \({(\mat{M} + \mat{M}\T)/2}\), so many matrices give the same form, and a formula that uses \(\mat{M}\) alone can be wrong.

“The order of the letters in a subscript is cosmetic.”

The order of the factors in a term does not matter: \({a_{ij} b_{jk}} = {b_{jk} a_{ij}}\), because numbers commute. The order of the letters inside one subscript does: \({a_{ji} b_{jk}}\) is \((\mat{A}\T\mat{B})_{ik}\), not \((\mat{A}\mat{B})_{ik}\).

“A letter may appear as often as I like.”

At most twice per term. \({a_{ij} x_j y_j}\) is not defined by the convention; if you mean \(\sum_j a_{ij} x_j y_j\), write the sum. And every term must carry the same free letters: in \({y_i = a_{ij} x_j + b_k}\) the last term has the wrong free index.

Check yourself

Recall

1. In \({y_i = a_{ij} x_j}\), which index is free and which is the dummy?

Show answer

\(i\) is free (once on each side, it names the entry). \(j\) is the dummy (twice, summed).

2. What does \({a_{ij} \delta_{jk}}\) equal, and why?

Show answer

\(a_{ik}\). In the sum over \(j\) only the term \(j = k\) is non-zero, and there \(\delta_{kk} = 1\).

3. Write the trace of \(\mat{A}\mat{B}\) in index notation.

Show answer

\({a_{ij} b_{ji}}\), summed over both \(i\) and \(j\).

4. Why is \({y_i = a_{jk} x_k}\) not a valid equation?

Show answer

Its free indices do not match: \(i\) on the left, \(j\) on the right.

5. What is the gradient of \(\vect{x}\T \mat{S} \vect{x}\) for any square \(\mat{S}\)?

Show answer

\({(\mat{S} + \mat{S}\T)\vect{x}}\), which equals \(2\mat{S}\vect{x}\) only when \(\mat{S}\) is symmetric.

Exercises

1. ★ Write in index notation and name the free indices: \(\vect{y} = \mat{A}\T\vect{x}\); the dot product \(s = \vect{u} \cdot \vect{v}\); and \(\mat{D} = \mat{A}\mat{B}\mat{C}\).

Show solution

\({y_i = a_{ji} x_j}\), free index \(i\). \({s = u_i v_i}\), no free index: a single number. \({d_{il} = a_{ij} b_{jk} c_{kl}}\), free indices \(i\) and \(l\), dummies \(j\) and \(k\).

2. ★★ Prove \(\operatorname{tr}(\mat{A}\mat{B}\mat{C}) = \operatorname{tr}(\mat{C}\mat{A}\mat{B})\) in index notation.

Show solution

By Exercise 1, \((\mat{A}\mat{B}\mat{C})_{il} = a_{ij} b_{jk} c_{kl}\), so the trace sets \(l = i\):

\[ \begin{aligned} \operatorname{tr}(\mat{A}\mat{B}\mat{C}) &= a_{ij}\, b_{jk}\, c_{ki} \\ &= c_{ki}\, a_{ij}\, b_{jk} \\ &= (\mat{C}\mat{A}\mat{B})_{kk} . \end{aligned} \]

The middle line reorders three numbers. In the last, the dummy \(i\) joins \(\mat{C}\) to \(\mat{A}\) and \(j\) joins \(\mat{A}\) to \(\mat{B}\), and the outer letters are both \(k\): a trace.

3. ★★★ Show in index notation that the gradient of \(g(\vect{x}) = \norm{\mat{A}\vect{x}}^2\) is \(2\mat{A}\T\mat{A}\vect{x}\), and evaluate it for the running \(\mat{A}\) at \(\vect{x} = (1, 1)\).

Show solution

\((\mat{A}\vect{x})_i = a_{ij} x_j\), so \(g = a_{ij} x_j\, a_{il} x_l\) (a second dummy \(l\), so that no letter appears more than twice). Differentiate with the product rule:

\[ \begin{aligned} \pd{g}{x_k} &= a_{ij}\, \delta_{jk}\, a_{il} x_l + a_{ij} x_j\, a_{il}\, \delta_{lk} \\ &= a_{ik}\, a_{il} x_l + a_{ij} x_j\, a_{ik} \\ &= 2\, a_{ik}\, a_{il}\, x_l . \end{aligned} \]

The last line renames \(j\) as \(l\). Since \(a_{ik} = (\mat{A}\T)_{ki}\), this is \(2(\mat{A}\T\mat{A}\vect{x})_k\). For the running \(\mat{A}\), \(\mat{A}\T\mat{A}\) has rows \((20, 10)\) and \((10, 10)\), so the gradient at \((1, 1)\) is \(2\,(30, 20) = (60, 40)\). Check: \(g = {(4x_1 + x_2)^2} + {(2x_1 + 3x_2)^2}\), and \(\pd{g}{x_1} = {8 \cdot 5} + {4 \cdot 5} = 60\), \(\pd{g}{x_2} = {2 \cdot 5} + {6 \cdot 5} = 40\).

Summary card

A repeated index is summed; a single index names the answer.

The anchor figure again: matrices A and B arranged around their product C, with row 1 of A and column 2 of B shaded, the shared letter j marked on both, and c sub 1 2 equals 1 times 0 plus 2 times 1 plus 3 times 2 equals 8.
M03.10.2-F1

\(c_{ik} = a_{ij}\, b_{jk}\).

  • Convention: twice in a term means summed; at most twice; free indices match on both sides.
  • Delta: \({a_{ij} \delta_{jk}} = a_{ik}\), \(\delta_{ii} = n\), \(\pd{x_i}{x_k} = \delta_{ik}\).
  • Identities: \((\mat{A}\mat{B})\T = \mat{B}\T\mat{A}\T\), \(\operatorname{tr}(\mat{A}\mat{B}) = \operatorname{tr}(\mat{B}\mat{A})\).
  • Gradient: \({\nabla(x_i s_{ij} x_j)} = {(\mat{S} + \mat{S}\T)\vect{x}}\).

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • Eric W. Weisstein, Einstein Summation, MathWorld: repeated indices are implicitly summed, an index appears at most twice in a term, free indices must match (with invalid examples of both kinds), and the convention comes from Einstein’s 1916 paper on general relativity, section 5.
  • Marc Peter Deisenroth, A. Aldo Faisal and Cheng Soon Ong, Mathematics for Machine Learning, section 5.4 (gradients of matrices are tensors, thought of as multidimensional arrays) and section 5.5, identity (5.107): the derivative of \(\vect{x}\T\mat{B}\vect{x}\) is \({\vect{x}\T(\mat{B} + \mat{B}\T)}\), the row-vector form of Result 4.
  • Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, section 2.1: entries of arrays written with one index per axis, \(A_{i,j,k}\), and the transpose as the mirror image across the main diagonal.

Next. M03.10.3 einsum: any contraction from scratch turns an index expression into a string such as ij,jk->ik and computes it.

This lesson unlocks

M03.10.3 einsum: any contraction from scratch M05.7 Matrix calculus identities

Your progress is saved in this browser only. No account, no tracking.
Back to top