\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

M08 · Optimization

A free, illustrated L2 to L3 course of 11 modules in the Mathematics track. Modules include: Optimization problems and their geometry; Convex sets and convex functions; Gradient descent and step sizes.

M08 L2 to L3 · Intermediate to Advanced 11 modules · about 44 lessons Planned · Ring 3

Part of the Mathematics track. Ships in Ring 3 (Breadth) of the roadmap.

Modules

1. Optimization problems and their geometry
2. Convex sets and convex functions
3. Gradient descent and step sizes
4. Momentum, Nesterov and adaptive methods
5. Newton and quasi-Newton methods
6. Stochastic gradient descent and variance
7. Constrained optimization and Lagrange multipliers
8. KKT conditions and duality
9. Linear and quadratic programming
10. Proximal methods and non-smooth problems
11. Non-convex landscapes: saddles, plateaus and minima

Signature illustrations

The pictures a learner should still see in their head a year later:

Optimizer paths on contour plots, convexity chord test, Lagrange tangency picture, duality gap diagram, escaping a saddle.

Status

This course is planned, not written yet. It ships with Ring 3 (Breadth).

Every lesson will follow the same twelve-part blueprint with at least three purpose-made figures. Watch the repository to hear when modules ship, or help write and review it.