\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

M04.4 · Optimization in one dimension

A free, illustrated module of 4 lessons and one interactive in M04 Calculus I: Single Variable: Critical points and the first-derivative test; The second derivative and convexity; Optimisation problems solved exactly; Gradient descent and Newton’s method in one dimension.

M04.4 L1 · Core 4 lessons · about 100 min Live · in review

Part of M04 Calculus I: Single Variable in the Mathematics track.

Training a model means finding the lowest point of a loss. These four lessons show how the slope and the curvature of a function locate its peaks and valleys: the sign of the derivative says where a curve rises and falls, the second derivative tells a valley from a peak and marks out the convex functions whose only valley is the lowest point, and the recipe of critical points and endpoints solves problems exactly, from the mean to a coin’s maximum likelihood. When no formula will do, gradient descent and Newton’s method search for the bottom step by step.

What you need first. Limits and one-sided limits (M04.1), the derivative as a slope and the linear approximation (M04.2), and the rules of differentiation with the chain rule (M04.3). The code uses NumPy and a little SciPy, explaining each call.

Lessons

1. Critical points and the first-derivative test M04.4.1 25 min

   A function rises where its derivative is positive and falls where it is negative. Use that to find where a curve can turn, prove that a smooth peak or valley has slope zero, and tell peaks from valleys by how the sign of the slope changes.

2. The second derivative and convexity M04.4.2 25 min

   The second derivative measures how fast the slope changes: acceleration for a moving cart, bending for a graph. Use it to tell peaks from valleys in one step, find inflection points, and prove that the flat point of a convex function, such as a squared-error loss, is its lowest point.

3. Optimisation problems solved exactly M04.4.3 25 min

   A recipe for exact optimisation: write the quantity as a function of one variable, find its critical points, and compare them with the ends of the interval. Then derive two results data science uses daily: the mean minimises squared deviations, and 7 heads in 10 tosses make 0.7 the most likely chance of heads.

4. Gradient descent and Newton’s method in one dimension M04.4.4 25 min

   When f’(x) = 0 cannot be solved by hand, search for the minimum. Derive gradient descent from the tangent line and Newton’s method from the tangent line of the slope, run both on x squared plus e to the minus x, compare them with SciPy, and see how the start decides which valley you reach.

The interactive

M04.4-X1 · Descent explorer. Drag a ball along the running cubic, the double well or the running loss, with its tangent and the signs of the first and second derivative marked. Take gradient-descent steps with a learning rate you choose, or Newton steps, and see which bottom each start reaches and when a step overshoots. It lives in Gradient descent and Newton’s method in one dimension.