\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

S05 · Experiment Design and A/B Testing

A free, illustrated L2 course of 8 modules in the Statistics and Data Analysis track. Modules include: Randomization and controlled experiments; Metrics and guardrails; Power and test duration.

S05 L2 · Intermediate 8 modules · about 32 lessons Planned · Ring 3

Part of the Statistics and Data Analysis track. Ships in Ring 3 (Breadth) of the roadmap.

Modules

1. Randomization and controlled experiments
2. Metrics and guardrails
3. Power and test duration
4. Analysis and variance reduction
5. Sequential testing and the peeking problem
6. Bandit experiments
7. Interference and network effects
8. Experimentation platforms and practice

Signature illustrations

The pictures a learner should still see in their head a year later:

Diagrams of randomization units, power curves, false-positive inflation from peeking, variance before and after adjustment.

Status

This course is planned, not written yet. It ships with Ring 3 (Breadth).

Every lesson will follow the same twelve-part blueprint with at least three purpose-made figures. Watch the repository to hear when modules ship, or help write and review it.