\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

A02 · Large Language Models

A free, illustrated L3 course of 11 modules in the Applied AI track. Modules include: Anatomy and scale of an LLM; Data: sources, filtering and mixing; Pretraining at scale.

A02 L3 · Advanced 11 modules · about 44 lessons Planned · Ring 3

Part of the Applied AI track. Ships in Ring 3 (Breadth) of the roadmap.

Modules

1. Anatomy and scale of an LLM
2. Data: sources, filtering and mixing
3. Pretraining at scale
4. Supervised fine-tuning
5. Parameter-efficient tuning
6. Post-training with human feedback and verifiable rewards
7. Decoding and sampling
8. Inference: caching, quantization and batching
9. Evaluation and benchmarks
10. Reasoning models
11. Build and train a small LLM from scratch

Signature illustrations

The pictures a learner should still see in their head a year later:

A map of the training pipeline, the low-rank update picture, token distributions under temperature and top-p, the key-value cache over time, a taxonomy of evaluations.

Status

This course is planned, not written yet. It ships with Ring 3 (Breadth).

Every lesson will follow the same twelve-part blueprint with at least three purpose-made figures. Watch the repository to hear when modules ship, or help write and review it.