\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

S02.5 · Hypothesis tests and p-values

A free, illustrated module of 4 lessons and one interactive in S02 Statistical Inference: The logic of a test; The p-value; Decisions and the two kinds of error; Tests and intervals, significance and importance.

S02.5 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of S02 Statistical Inference in the Statistics and Data Analysis track.

A hypothesis test assumes the dullest explanation of the data and asks how surprising the data would be if it were true. These four lessons build the test from that idea: the null and alternative hypotheses, a statistic and its null distribution, worked exactly for 7 heads in 10 tosses and for a classifier’s 90 of 100 against a baseline; the p-value one- and two-sided, the z-test, and the proof that under the null a p-value is uniform; a level fixed in advance, the two kinds of error drawn as an overlap diagram, and the same trade in a classifier’s threshold; and finally the duality of tests and confidence intervals, the gap between significant and important, and the false positives that many analyses produce.

What you need first. Sampling distributions and the standard error (S02.1), the binomial and normal families (M06.8), and the central limit theorem (M06.11). The last lesson uses the t-interval of S02.4.

Lessons

1. The logic of a test S02.5.1 25 min

   A hypothesis test assumes a dull explanation, works out what data it would produce, and asks how surprising the real data are. Count the exact chance of 7 or more heads in 10 fair tosses, then test a classifier’s 90 of 100 against a baseline of 0.85.

2. The p-value S02.5.2 25 min

   The p-value is the chance, if the null hypothesis were true, of a result at least as extreme as the one observed. Compute it one- and two-sided, run a z-test with math.erf, prove that under the null it is uniform, and learn what it does not measure.

3. Decisions and the two kinds of error S02.5.3 25 min

   Fix a level alpha before the data arrive and a p-value becomes a yes-or-no decision. Draw the overlap diagram of the null and alternative distributions, read off the Type I and Type II error rates, and see the same trade in a classifier’s threshold.

4. Tests and intervals, significance and importance S02.5.4 25 min

   A two-sided test rejects a null value exactly when it lies outside the confidence interval, so an interval is the set of values the data do not reject. Then separate statistical from practical significance, and see how running many analyses inflates false positives.

The interactive

S02.5-X1 · p-value machine. The binomial null distribution for n tosses of a fair coin, with the observed count on a slider: the bars at least as extreme fill in, one- or two-sided, and the read-out gives the p-value. A second panel runs a thousand experiments under the null and stacks their p-values into a histogram, lumpy for a few tosses and flat for the z-test, with the share at or below 0.05 marked. It lives in The p-value.