Skip to main content

The directional derivative

Walk from a point in any direction and the heights you pass make an ordinary curve. Its slope is the directional derivative. For a differentiable function it is one dot product, the gradient times the direction, in any number of dimensions.
M05.3.1L2 Intermediate25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: M05.1 Functions of several variables and level sets M05.2 Partial derivatives and the gradient M04.10 Partial derivatives, a first look

What you will be able to do
  • Define the directional derivative of a function of n inputs along a unit vector as the slope of a slice at zero.
  • Derive the formula slope equals gradient dot direction from the linear approximation of a differentiable function.
  • Read a directional derivative as the signed length of the gradient’s shadow on the direction, and compute it in any dimension.
  • Explain with x squared y over (x to the fourth plus y squared) why slopes in every direction do not make a function differentiable.
  • Check a directional derivative against a central difference along the direction.

The question

A model with a million weights takes a training step: every weight moves at once, in one particular direction. How fast does the loss change as the model sets off that way? The partial derivatives give the slope along each axis, one weight at a time. Can they also give the slope along a direction that is no axis at all?

Intuition

Stand at a point \(\vect{a}\) on the map of a function \(f\) and choose a direction, a vector \(\vect{u}\) of length 1. Walk along the straight line \({\vect{a} + s\,\vect{u}}\). The heights you pass make an ordinary function of one number, the slice \({g(s) = f(\vect{a} + s\,\vect{u})}\) of M05.1.3. Its slope where you stand, at \({s = 0}\), is the slope of \(f\) in the direction \(\vect{u}\): the directional derivative, written \({D_{\vect{u}} f(\vect{a})}\).

Two panels. The left panel, titled On the map: a line through a, shows blue contour ellipses of the running bowl with x and y from minus 0.6 to 2.4. An orange point sits at (1, 1) on a thick blue ellipse, an orange arrow labelled u leaves it up and to the right, and a dashed blue line labelled the line a plus s u runs through the point along the arrow. The right panel, titled Along the line: the slice g, plots the blue curve g of s equals 3 plus 4.4 s plus 1.64 s squared for s from minus 1.2 to 1.2. An orange point marks g of 0 equals f of (1, 1) equals 3, and a green tangent line through it is labelled slope 4.4. A sentence underneath reads: the slope of f along u is the slope of the slice at s equals 0.
M05.3.1-F1

The running bowl \({f(x, y) = x^2 + 2y^2}\) at \({\vect{a} = (1, 1)}\), in the direction \({\vect{u} = (0.6, 0.8)}\). On the map the direction is a line through the point (left). Along that line \(f\) is an ordinary curve (right), and its slope at \({s = 0}\) is the slope of \(f\) in that direction, 4.4.

For the running bowl the slice along \({\vect{u} = (0.6, 0.8)}\) from \({(1, 1)}\) is a parabola, \({g(s) = 3 + 4.4s + 1.64s^2}\), so the slope is 4.4. The two partial derivatives of M05.2 are the special cases where \(\vect{u}\) points along an axis. This lesson shows that you never need to build the slice: the gradient already knows the slope in every direction, and one dot product reads it off.

Build it up

Step 1. Why the direction has length 1. Walk along \({2\vect{u}}\) instead of \(\vect{u}\) and you cover the same line twice as fast, so the slope per unit of \(s\) doubles. A slope should belong to the direction alone, so we measure \(s\) as distance on the map. That needs \({\norm{\vect{u}} = 1}\): a unit vector. The direction \({(3, 4)}\) becomes \({(3, 4)/5 = (0.6, 0.8)}\), since \({\sqrt{9 + 16} = 5}\).

Step 2. The slice, by hand. On the line, \({x = 1 + 0.6s}\) and \({y = 1 + 0.8s}\). Put them into \({x^2 + 2y^2}\) and expand each square:

\[ \begin{aligned} g(s) &= (1 + 0.6s)^2 + 2(1 + 0.8s)^2 \\ &= 1 + 1.2s + 0.36s^2 \\ &\quad + 2 + 3.2s + 1.28s^2 \\ &= 3 + 4.4s + 1.64s^2 . \end{aligned} \]

Step 3. Rise over step. Divide the rise \({g(s) - g(0)}\) by the step \(s\):

\[ \begin{aligned} \frac{g(s) - g(0)}{s} &= \frac{4.4s + 1.64s^2}{s} \\ &= 4.4 + 1.64s . \end{aligned} \]

For \({s = 1}\), 0.5 and 0.25 this is 6.04, 5.22 and 4.81. As the step shrinks, the secant lines turn into the tangent and the ratio settles on 4.4.

Two panels. The left panel, titled Secants along u, shows the blue slice curve against the step s, with an orange point at s equals 0, three vermillion secant lines from it to points at s equals 0.25, 0.5 and 1, the steepest labelled secant to s equals 1, and a dashed green tangent line labelled tangent. The right panel, titled Rise over step, plots the slope of each secant against s from minus 1 to 1: vermillion dots on a straight line, labelled 2.76 at s equals minus 1, then 4.81, 5.22 and 6.04 at s equals 0.25, 0.5 and 1, and a green ring at s equals 0 labelled limit 4.4.
M05.3.1-F2

The secant from \({s = 0}\) to a nearby point has slope \({4.4 + 1.64s}\). The shorter the step, the closer it is to 4.4, the slope of the tangent.

Step 4. Where 4.4 comes from. At \({(1, 1)}\) the gradient of the bowl is \({(2x, 4y) = (2, 4)}\). Multiply it entry by entry with the direction and add: \({2 \times 0.6 + 4 \times 0.8} = {1.2 + 3.2} = {4.4}\). The same number. The next section shows that this is no accident.

The formal statement

NoteDefinition: directional derivative

Let \({f : \R^n \to \R}\), let \(\vect{a}\) be a point and \(\vect{u}\) a unit vector. The directional derivative of \(f\) at \(\vect{a}\) in the direction \(\vect{u}\) is

\[ D_{\vect{u}} f(\vect{a}) = \lim_{s \to 0} \frac{f(\vect{a} + s\vect{u}) - f(\vect{a})}{s}, \]

when the limit exists. It is \({g'(0)}\) for the slice \({g(s) = f(\vect{a} + s\vect{u})}\).

Along the axis \(\vect{e}_i\) only input \(i\) moves, so \({D_{\vect{e}_i} f = \pd{f}{x_i}}\): every partial derivative is a directional derivative.

NoteResult: the slope is a dot product

If \(f\) is differentiable at \(\vect{a}\), then for every unit vector \(\vect{u}\)

\[ \begin{aligned} D_{\vect{u}} f(\vect{a}) &= \cgrad{\nabla f(\vect{a})}\T\cparam{\vect{u}} \\ &= \sum_{i=1}^{n} \pd{f}{x_i}(\vect{a})\,u_i . \end{aligned} \]

Proof. Differentiable at \(\vect{a}\) means (M05.2.2) that for every small step \(\vect{h}\)

\[ \begin{aligned} f(\vect{a} + \vect{h}) &= f(\vect{a}) + \nabla f(\vect{a})\T\vect{h} \\ &\quad + r(\vect{h}), \end{aligned} \]

with a remainder whose ratio \({r(\vect{h}) / \norm{\vect{h}}}\) tends to 0 as \(\vect{h}\) shrinks to \(\vect{0}\). Take the step \({\vect{h} = s\vect{u}}\) with \({s \neq 0}\). Move \({f(\vect{a})}\) to the left and divide by \(s\):

\[ \begin{aligned} &\frac{f(\vect{a} + s\vect{u}) - f(\vect{a})}{s} \\ &\quad = \nabla f(\vect{a})\T\vect{u} + \frac{r(s\vect{u})}{s} . \end{aligned} \]

The first term on the right does not depend on \(s\). For the second, \({\norm{s\vect{u}} = \abs{s}}\) because \(\vect{u}\) has length 1, so \({\abs{r(s\vect{u}) / s}} = {\abs{r(s\vect{u})} / \norm{s\vect{u}}}\). As \({s \to 0}\) the step \({s\vect{u}}\) shrinks to \(\vect{0}\) and this ratio tends to 0. So the left side has a limit, and it is \({\nabla f(\vect{a})\T\vect{u}}\). \(\square\)

The picture. Because \({\norm{\vect{u}} = 1}\), the dot product \({\nabla f\T\vect{u}}\) is the signed length of the shadow that the gradient casts on the line of \(\vect{u}\) (the scalar projection of M02.3.4). A long shadow means a steep climb; a shadow pointing backwards means descent.

Two panels on a faint grid, each with an orange point at the origin, a green arrow for the gradient (2, 4) and a short orange arrow u. In the left panel u equals (0.6, 0.8); a grey line runs along u, a dashed black line drops from the tip of the gradient onto it at a right angle, and the part of the line from the point to the foot is shaded green and labelled shadow equals 4.4. In the right panel u equals (1, 0); the gradient's tip drops straight down onto the horizontal line, and the green shadow is labelled shadow equals 2 equals partial f over partial x. A sentence underneath: D u f equals gradient dot u, the signed length of the gradient's shadow on the line of u.
M05.3.1-F3

The directional derivative is the gradient’s shadow on the direction. Along \({(0.6, 0.8)}\) the shadow of \({(2, 4)}\) is 4.4 long; along the \(x\) axis it is 2, the partial derivative in \(x\).

Two consequences follow at once. Reversing the direction reverses the slope, \({D_{-\vect{u}} f = -D_{\vect{u}} f}\), since the dot product is linear. And the slope in a mixed direction is a weighted sum of the partial derivatives, with the entries of \(\vect{u}\) as weights.

Some books, Boyd and Vandenberghe among them, call \({\nabla f\T\vect{v}}\) the directional derivative for any vector \(\vect{v}\), not only a unit one. That number is the rate of change per unit of \(s\) along \({\vect{a} + s\vect{v}}\), which is \({\norm{\vect{v}}}\) times our slope.

Go deeper: a formula that works in every direction is still not enough

Let \({h(x, y) = 1}\) on the parabola \({y = x^2}\) (except at the origin) and \({h = 0}\) everywhere else. A line \({y = mx}\) through the origin meets the parabola only where \({x^2 = mx}\), that is at \({x = 0}\) and \({x = m}\), and the axis \({x = 0}\) meets it only at the origin. So on every line, near the origin, \(h\) is 0: every directional derivative at the origin exists and equals 0, which is \({\nabla h(\vect{0})\T\vect{u}}\) with \({\nabla h(\vect{0}) = \vect{0}}\). Yet \(h\) is not even continuous there, since it equals 1 at points of the parabola as close to the origin as you like. Lines cannot see what happens along curves; differentiability is a statement about every way of approaching the point at once.

Worked example

The bowl at \({(1, 1)}\) along \({(3, 4)/5}\). Route 1 builds the slice, as in Step 2, and differentiates it: \({g'(s) = 4.4 + 3.28s}\), so \({g'(0) = 4.4}\). Route 2 takes the gradient \({(2, 4)}\) and the unit direction \({(0.6, 0.8)}\) and forms one dot product, \({1.2 + 3.2 = 4.4}\).

A diagram of boxes in two columns. The left column, headed Route 1: the slice, has three boxes joined by downward arrows: g of s equals f of (1 plus 0.6 s, 1 plus 0.8 s); then, after expand the squares, g of s equals 3 plus 4.4 s plus 1.64 s squared; then, after differentiate in s, g prime of s equals 4.4 plus 3.28 s. The right column, headed Route 2: the formula, has boxes for the gradient at (1, 1) equals (2, 4), for u equals (0.6, 0.8), and, after dot product, 2 times 0.6 plus 4 times 0.8. Green arrows from both columns meet in a yellow box at the bottom: D u f at (1, 1) equals 4.4, one labelled set s equals 0 and the other 1.2 plus 3.2.
M05.3.1-F4

Two routes to the same slope. The slice needs algebra that grows with the function; the formula needs only the gradient.

The running loss at the origin. From M05.2.3 the gradient of the running loss is \({(15w + 5b - 23.5,\; 5w + 2b - 8)}\), which is \({(-23.5, -8)}\) at \({(0, 0)}\). Along the \(w\) axis the slope is \(-23.5\); along the \(b\) axis it is \(-8\). The diagonal unit vector \({(1, 1)/\sqrt{2}}\) gives

\[ \frac{-23.5 - 8}{\sqrt{2}} = \frac{-31.5}{\sqrt{2}} \approx -22.27 . \]

Straight towards the lowest point \({(1.4, 0.5)}\), whose distance from the origin is \({\sqrt{1.96 + 0.25} = \sqrt{2.21}}\), the slope is

\[ \begin{aligned} &\frac{-23.5 \times 1.4 - 8 \times 0.5}{\sqrt{2.21}} \\ &\quad = \frac{-36.9}{\sqrt{2.21}} \approx -24.82 . \end{aligned} \]

Backwards, along \({(-1, 0)}\), the loss climbs at 23.5. As a check on the first number, the slice of the loss along the \(w\) axis is \({7.5w^2 - 23.5w + 18.5}\), whose slope at 0 is \(-23.5\).

A contour map of the running loss with w across from minus 1 to 1.9 and b up from minus 1 to 1.25, reddish-purple contour lines, a thick one labelled L equals 18.5 through an orange point at the origin, and a black star at (1.4, 0.5). Six orange unit arrows leave the origin, each tagged in green with its slope: minus 23.5 to the right, minus 8 upwards, minus 22.27 along the diagonal, 23.5 to the left, 8 downwards, and minus 24.82 on a longer arrow aimed at the star. Text beside the map says the arrow aimed at the star has slope minus 24.82, very nearly the steepest of all.
M05.3.1-F5

Slopes of the running loss at \({(0, 0)}\), each one \({\nabla L\T\vect{u}}\). Heading for the lowest point is almost the steepest way down from here; lesson 2 finds the steepest exactly.

Three inputs. For \({f(x, y, z) = x^2 + 2y^2 + 3z^2}\) at \({(1, 1, 1)}\) the gradient is \({(2x, 4y, 6z) = (2, 4, 6)}\). The direction \({(1, 2, 2)}\) has length \({\sqrt{1 + 4 + 4} = 3}\), so \({\vect{u} = (1, 2, 2)/3}\) and

\[ D_{\vect{u}} f = \frac{2 + 8 + 12}{3} = \frac{22}{3} \approx 7.33 . \]

No picture of a function of three inputs is needed; the dot product works the same in any dimension.

Code

From scratch: the formula against a central difference taken along the direction, which needs only two values of \(f\), whatever the number of inputs.

import numpy as np

def directional(f, a, u, h=1e-6):
    """Slope of f at a along u, by a central difference along the line a + s u."""
    u = u / np.linalg.norm(u)                    # a unit vector, always
    return (f(a + h * u) - f(a - h * u)) / (2 * h)

bowl = lambda p: p[0] ** 2 + 2 * p[1] ** 2
loss = lambda p: 7.5 * p[0] ** 2 + 5 * p[0] * p[1] + p[1] ** 2 - 23.5 * p[0] - 8 * p[1] + 18.5
bowl3 = lambda p: p[0] ** 2 + 2 * p[1] ** 2 + 3 * p[2] ** 2

cases = [("bowl", bowl, [1, 1], [3, 4], [2, 4]),
         ("loss", loss, [0, 0], [1.4, 0.5], [-23.5, -8]),
         ("bowl in 3-D", bowl3, [1, 1, 1], [1, 2, 2], [2, 4, 6])]
for name, f, a, v, grad in cases:
    a, v, grad = np.array(a, float), np.array(v, float), np.array(grad, float)
    print(f"{name:12s} formula {grad @ v / np.linalg.norm(v):9.4f}   difference {directional(f, a, v):9.4f}")
bowl         formula    4.4000   difference    4.4000
loss         formula  -24.8216   difference  -24.8216
bowl in 3-D  formula    7.3333   difference    7.3333

The same check on a function of ten inputs, at a random point and in a random direction. Its gradient was worked out by hand: \({2 i\,x_i}\) from each term \({i\,x_i^2}\), plus the sine term’s \({\cos(x_1 x_2)\,(x_2, x_1)}\) in the first two places.

rng = np.random.default_rng(1)
n = 10
weights = np.arange(1, n + 1)
f10 = lambda x: np.sum(weights * x ** 2) + np.sin(x[0] * x[1])

def grad10(x):
    g = 2 * weights * x
    g[0] += np.cos(x[0] * x[1]) * x[1]
    g[1] += np.cos(x[0] * x[1]) * x[0]
    return g

a, v = rng.normal(size=n), rng.normal(size=n)
u = v / np.linalg.norm(v)
print("formula   ", grad10(a) @ u)
print("difference", directional(f10, a, u))
formula    -5.73997472759745
difference -5.739974728413699

The two agree to six decimal places or better (the exact digits depend on rounding). Now the function of the first pitfall below: the difference quotients along the diagonal settle on \({1/\sqrt{2} \approx 0.707}\), while the formula with its gradient \({(0, 0)}\) would say 0.

def h(x, y):
    return 0.0 if x == 0 and y == 0 else x * x * y / (x ** 4 + y * y)

d = np.array([1.0, 1.0]) / np.sqrt(2)
for s in (1e-1, 1e-3, 1e-5):
    print(f"s = {s:g}:  rise over step {h(s * d[0], s * d[1]) / s:.6f}")
print("on the parabola y = x^2:", [h(x, x * x) for x in (0.1, 0.01, 0.001)])
s = 0.1:  rise over step 0.703589
s = 0.001:  rise over step 0.707106
s = 1e-05:  rise over step 0.707107
on the parabola y = x^2: [0.5000000000000001, 0.5, 0.4999999999999999]

Explore

This module’s interactive, the direction dial, lives in lesson 3. Drag the point on a contour map and turn the direction slider: the side panel plots \({D_{\vect{u}} f}\) against the angle, and you can watch the slope of this lesson change as the direction turns.

Pitfalls

Two contour maps of x squared y over (x to the fourth plus y squared) for x and y from minus 1 to 1, whose contour lines are parabolas through the origin. The left map, marked with a cross and titled Lines say: smooth, has eight orange straight lines through a black point at the origin, labelled every line: a slope. The right map, marked with a tick and titled Parabola: a jump, highlights the thick blue parabola y equals x squared, labelled f equals one half, running into the black point at the origin, labelled f of (0, 0) equals 0. A note says contours are parabolas. A sentence underneath: a slope every way, yet not continuous at 0.
M05.3.1-F6

Wrong beside right. Every straight line through the origin gives the function a slope there, yet along the parabola \({y = x^2}\) it stays at \(\tfrac12\) all the way in, while at the origin it is 0.

“It has a slope in every direction, so it is differentiable.”

Take \({f(x, y) = x^2 y / (x^4 + y^2)}\) with \({f(0, 0) = 0}\). Along \({\vect{u} = (a, b)}\) with \({b \neq 0}\),

\[ \begin{aligned} \frac{f(sa, sb)}{s} &= \frac{s^3 a^2 b}{s\,(s^4 a^4 + s^2 b^2)} \\ &= \frac{a^2 b}{s^2 a^4 + b^2} \;\to\; \frac{a^2}{b} , \end{aligned} \]

where the second line cancels \({s^3}\) from top and bottom. Along the \(x\) axis (\({b = 0}\)) \(f\) is 0, so the slope is 0. Every directional derivative exists, but both partial derivatives are 0, so the formula would give 0 in every direction, which is wrong (along the diagonal it is \({1/\sqrt{2}}\)). Worse, on \({y = x^2}\) we get \({f = x^4 / (2x^4) = \tfrac12}\), so \(f\) is not even continuous at the origin. The formula \({\nabla f\T\vect{u}}\) needs differentiability.

“Any arrow will do as the direction.”

With \({\vect{v} = (3, 4)}\) instead of \({(0.6, 0.8)}\), \({\nabla f\T\vect{v} = 6 + 16 = 22}\), five times the true slope 4.4, because \(\vect{v}\) is five units long. Divide by \({\norm{\vect{v}}}\) first, or you are measuring the speed of a walker, not the steepness of the ground.

“The slope tells me how much the loss changes on a step.”

Only for a short step. On the bowl a step of length 1 along \({(0.6, 0.8)}\) raises \(f\) by \({g(1) - g(0) = 6.04}\), not 4.4: the slope describes the tangent, and the curve bends away from it. This is why gradient methods take small steps, controlled by a learning rate (ML01.5).

Check yourself

Recall

1. What is \({D_{\vect{u}} f(\vect{a})}\), in words?

Show answer

The slope of \(f\) at \(\vect{a}\) when you move along the unit vector \(\vect{u}\): the derivative at \({s = 0}\) of the slice \({g(s) = f(\vect{a} + s\vect{u})}\).

2. What is the formula for \({D_{\vect{u}} f}\) when \(f\) is differentiable?

Show answer

\({D_{\vect{u}} f(\vect{a})} = {\nabla f(\vect{a})\T\vect{u}}\), the sum of the partial derivatives weighted by the entries of \(\vect{u}\).

3. Why must \(\vect{u}\) have length 1?

Show answer

So that \(s\) measures distance. A direction of length \(k\) would multiply the slope by \(k\).

4. What is \({D_{\vect{e}_i} f}\)?

Show answer

The partial derivative \({\pd{f}{x_i}}\): moving along the axis \(\vect{e}_i\) changes input \(i\) only.

5. Give a function that has a slope in every direction at a point but is not differentiable there.

Show answer

\({f(x, y) = x^2 y / (x^4 + y^2)}\) with \({f(0, 0) = 0}\), at the origin. Its slope along \({(a, b)}\) is \({a^2/b}\) (0 when \({b = 0}\)), not of the form \({\nabla f\T\vect{u}}\), and it equals \(\tfrac12\) on the parabola \({y = x^2}\).

Exercises

1. ★ Find the slope of the running bowl at \({(2, -1)}\) in the direction \({(0.6, -0.8)}\), by the formula and by the slice.

Show solution

The gradient is \({(2x, 4y) = (4, -4)}\), so the slope is \({4 \times 0.6 + (-4) \times (-0.8)} = {2.4 + 3.2} = {5.6}\). By the slice,

\[ \begin{aligned} g(s) &= (2 + 0.6s)^2 \\ &\quad + 2(-1 - 0.8s)^2 \\ &= 4 + 2.4s + 0.36s^2 \\ &\quad + 2 + 3.2s + 1.28s^2 \\ &= 6 + 5.6s + 1.64s^2 , \end{aligned} \]

so \({g'(0) = 5.6}\).

2. ★★ At \({(w, b) = (1, 0)}\) on the running loss, find the slope along \({(1, 1)/\sqrt{2}}\), and find a unit direction in which the slope is 0.

Show solution

The gradient is \({(15 - 23.5,\; 5 - 8) = (-8.5, -3)}\). Along the diagonal the slope is \({(-8.5 - 3)/\sqrt{2}} = {-11.5/\sqrt{2}} \approx {-8.13}\). A direction with slope 0 has zero dot product with \({(-8.5, -3)}\), for example \({(3, -8.5)}\), since \({-8.5 \times 3 + (-3)(-8.5) = 0}\). Its length is \({\sqrt{9 + 72.25} = \sqrt{81.25}}\), so the unit direction is \({(3, -8.5)/\sqrt{81.25}} \approx {(0.333, -0.943)}\).

3. ★★★ For \({f(\vect{x}) = \norm{\vect{x}}^2}\) on \(\R^n\), compute \({D_{\vect{u}} f(\vect{a})}\) from the definition, without the gradient. For which \(\vect{u}\) is it 0?

Show solution

Expand with the dot product (M02.3.2):

\[ \begin{aligned} \norm{\vect{a} + s\vect{u}}^2 &= \norm{\vect{a}}^2 + 2s\,\vect{a}\T\vect{u} \\ &\quad + s^2\norm{\vect{u}}^2 . \end{aligned} \]

Subtract \({\norm{\vect{a}}^2}\), use \({\norm{\vect{u}} = 1}\) and divide by \(s\): the ratio is \({2\vect{a}\T\vect{u} + s}\), which tends to \({2\vect{a}\T\vect{u}}\). This matches the formula with \({\nabla f = 2\vect{x}}\). It is 0 exactly when \(\vect{u}\) is perpendicular to \(\vect{a}\): moving at right angles to the radius, the length does not change to first order. At \({\vect{a} = \vect{0}}\) it is 0 in every direction.

Summary card

The slope of \(f\) along a unit direction is the gradient dotted with that direction.

The anchor figure again: on the bowl's contour map a line through the point (1, 1) along u equals (0.6, 0.8), and beside it the slice along that line, a parabola whose tangent at s equals 0 has slope 4.4.
M05.3.1-F1

Slope of the slice at zero.

  • Definition: \({D_{\vect{u}} f(\vect{a}) = g'(0)}\) for \({g(s) = f(\vect{a} + s\vect{u})}\), with \({\norm{\vect{u}} = 1}\).
  • Differentiable \(f\): \({D_{\vect{u}} f = \nabla f\T\vect{u}}\), the gradient’s shadow on \(\vect{u}\).
  • Axes give the partial derivatives; reversing \(\vect{u}\) reverses the slope.
  • Bowl at \({(1, 1)}\) along \({(0.6, 0.8)}\): 4.4. Loss at \({(0, 0)}\) towards the minimum: \(-24.82\).
  • Slopes in every direction do not imply differentiability.

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • Gilbert Strang and Edwin Herman, Calculus Volume 3 (OpenStax), Section 4.6: the directional derivative as a limit along a unit vector (Equation 4.36 in the plane, Equation 4.41 in space), Theorem 4.12 (it equals \({f_x\cos\theta + f_y\sin\theta}\) for a differentiable function) and Theorem 4.15 with Equation 4.42 (in three variables it is the gradient dotted with the unit vector).
  • Denis Auroux, MIT OpenCourseWare 18.02SC Multivariable Calculus, Session 38: Directional derivatives. The reading defines the directional derivative as the ratio of the change in \(w\) to the step length \(\Delta s\), proves that it is the gradient dotted with \(\vect{u}\) (its Equation 1) from the tangent plane approximation, and works an example with the direction \({3\vect{i} + 4\vect{j}}\) scaled to \({(3/5, 4/5)}\).
  • Marc Peter Deisenroth, A. Aldo Faisal and Cheng Soon Ong, Mathematics for Machine Learning, Section 5.2: the gradient of a function of \(n\) inputs collects its partial derivatives (Equation 5.40). The book writes it as a row of shape \({1 \times n}\); this course writes it as a column, so its products appear here with a transpose.
  • Stephen Boyd and Lieven Vandenberghe, Convex Optimization, Cambridge University Press, 2004 (free PDF from the authors), Section 9.4: it calls \({\nabla f(\vect{x})\T\vect{v}}\) the directional derivative of \(f\) at \(\vect{x}\) in the direction \(\vect{v}\), for any vector \(\vect{v}\), and uses it as the approximate change of \(f\) for a small step.

Next. M05.3.2 Steepest ascent and the gradient’s geometry asks which direction gives the largest slope, and why the gradient meets every level set at a right angle.

This lesson unlocks

M05.3.2 Steepest ascent and the gradient's geometry

Your progress is saved in this browser only. No account, no tracking.
Back to top