\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

RL05 · Model-Based RL and Planning

A free, illustrated L3 to L4 course of 8 modules in the Reinforcement Learning track. Modules include: Models, planning and control; Optimal control: LQR and model predictive control; Monte Carlo tree search.

RL05 L3 to L4 · Advanced to Expert 8 modules · about 32 lessons Planned · Ring 4

Part of the Reinforcement Learning track. Ships in Ring 4 (Expert) of the roadmap.

Modules

1. Models, planning and control
2. Optimal control: LQR and model predictive control
3. Monte Carlo tree search
4. From AlphaGo to AlphaZero
5. MuZero and learned models
6. World models and latent imagination
7. Model uncertainty and model exploitation
8. Model-based policy optimization

Signature illustrations

The pictures a learner should still see in their head a year later:

The four phases of tree search on a growing tree, imagined rollouts in latent space, compounding model error.

Status

This course is planned, not written yet. It ships with Ring 4 (Expert).

Every lesson will follow the same twelve-part blueprint with at least three purpose-made figures. Watch the repository to hear when modules ship, or help write and review it.