\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

S02.4 · Confidence intervals

A free, illustrated module of 4 lessons and one interactive in S02 Statistical Inference: What a confidence interval promises; The t-interval for a mean; Intervals for a proportion: test-set accuracy; Differences, and choosing n for a margin.

S02.4 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of S02 Statistical Inference in the Statistics and Data Analysis track.

A single estimate hides how far it could be from the truth; a confidence interval says. These four lessons turn the sampling distribution of the mean round into a range for the unknown mean, show in the dance of intervals what the 95 percent promises and what it does not, replace the unknown spread by the sample’s own with the t-interval, put an honest Wilson interval around a model’s test accuracy, and build intervals for differences between groups and the sample size a target margin needs.

What you need first. Sampling distributions and the standard error, and the t and chi-square distributions (S02.1), the central limit theorem (M06.11), and the variance of a difference (M06.10). The city of S01 and the two page designs are reused throughout.

Lessons

1. What a confidence interval promises S02.4.1 25 min

   Turn the sampling distribution of the mean round into a range for the unknown mean, watch a hundred such ranges dance around the truth, and learn exactly what the 95 percent does and does not promise.

2. The t-interval for a mean S02.4.2 25 min

   When the population’s spread is unknown, estimate it from the sample and widen the multiplier from 1.96 to a t quantile. Build the t-interval for design A’s five times, then test its promise by simulation on normal and on skewed data.

3. Intervals for a proportion: test-set accuracy S02.4.3 25 min

   Put an honest interval around a classifier’s test accuracy. See by exact calculation why the textbook Wald interval falls short near 0 and 1, derive the Wilson interval by solving a quadratic, and meet the exact Clopper-Pearson interval.

4. Differences, and choosing n for a margin S02.4.4 25 min

   Build confidence intervals for a difference of two means (pooled and Welch) and of two proportions from one rule, variances add, then turn the margin formula round to find how many observations or test examples a target precision needs.

The interactive

S02.4-X1 · The dance of confidence intervals. Draw samples one at a time or a hundred at once from the city’s rents or from a classifier with a known accuracy. Each sample’s interval joins a stack against the true value, misses turn the loss colour, and the read-out compares the running coverage with the promised level for the z, t, Wald and Wilson methods. It lives in What a confidence interval promises.