Skip to main content

Decisions and the two kinds of error

Fix a level alpha before the data arrive and a p-value becomes a yes-or-no decision. Draw the overlap diagram of the null and alternative distributions, read off the Type I and Type II error rates, and see the same trade in a classifier’s threshold.
S02.5.3L2 Intermediate25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: S02.5.2 The p-value

What you will be able to do
  • Turn a p-value into a decision with a level alpha fixed in advance, and write the rejection region of a one- or two-sided z-test.
  • Define Type I and Type II errors, and show that a level-alpha test makes a Type I error with probability alpha.
  • Derive the Type II error rate of the one-sided z-test for a given alternative (0.20 for the listing site against a true mean of 11).
  • Explain how moving the critical value trades one error for the other, and relate it to a classifier’s threshold.

The question

A spam filter must decide, an alarm must ring or stay quiet, a new model must ship or not. A p-value is a number between 0 and 1, not a decision. How do you turn it into a yes or a no, and how often will that yes or no be wrong?

From the last lesson

1. Define the p-value in one sentence.

Show answer

The probability, computed as if \(H_0\) were true, of a test statistic at least as extreme as the one observed.

2. What is the distribution of a continuous test’s p-value when \(H_0\) is true?

Show answer

Uniform on \({[0, 1]}\), so \({\Prob(P \le u) = u}\) for every \(u\).

Intuition

Go back to the listing site of the last lesson, whose 25 flats have a mean rent of 10.5 against the city’s 8.3. This time the question is one-sided: is the site dearer? Before looking, we promise to reject \({H_0 : \mu = 8.3}\) only if \(\bar{x}\) lands so high that a typical site would get there just 5 percent of the time. That fixes a line, and the picture shows what the line costs.

Two bell curves over an axis of sample means from 5 to 14. A black curve centred at 8.3 is labelled H0: mu = 8.3, and a sky-blue curve centred at 11 is labelled H1: mu = 11. An orange vertical line at 10.08 is labelled critical value 10.08, with keep H0 to its left and reject H0 to its right. The part of the black curve beyond the line is filled solid reddish purple and labelled Type I: alpha = 0.05. The part of the blue curve to the left of the line is hatched reddish purple and labelled Type II: beta = 0.20. The rest of the blue curve, beyond the line, is light green and labelled power 0.80.
S02.5.3-F1

The overlap diagram. If \(H_0\) is true, \(\bar{x}\) follows the black curve; if the site’s mean is really 11, it follows the blue one. The orange line splits the axis into “keep” and “reject”. The solid area is the chance of rejecting a true \(H_0\); the hatched area is the chance of keeping a false one.

Two curves, one line, two kinds of mistake. Slide the line right and the solid area shrinks while the hatched area grows. Slide it left and the reverse happens. No position removes both, because the curves overlap.

Build it up

Step 1. Choose a level in advance. The significance level \(\alpha\) is the largest chance of a false alarm you will accept, chosen before the data are seen. The usual choice is 0.05; a costly false alarm calls for 0.01. The decision rule is: reject \(H_0\) when the p-value is at most \(\alpha\), otherwise keep it.

Step 2. The rule as a region. For the one-sided z-test the p-value is \({1 - \Phi(z)}\), and

\[ \begin{aligned} 1 - \Phi(z) \le \alpha &\iff \Phi(z) \ge 1 - \alpha \\ &\iff z \ge z_{1-\alpha} . \end{aligned} \]

The last step holds because \(\Phi\) is increasing, and \({z_{1-\alpha}}\) is the quantile with \({\Phi(z_{1-\alpha}) = 1 - \alpha}\); for \({\alpha = 0.05}\) it is \({z_{0.95} = 1.645}\). The set of values that lead to rejection is the rejection region. On the scale of \(\bar{x}\) it starts at the critical value \({c = \mu_0 + z_{0.95}\,\sigma/\sqrt{n}}\), which is \({8.3 + 1.645 \times 1.082 = 10.08}\).

Step 3. Two-sided. The two-sided p-value is at most \(\alpha\) exactly when \({\abs{z} \ge z_{1-\alpha/2}}\), which is 1.96 for \({\alpha = 0.05}\). On the scale of \(\bar{x}\) we keep \(H_0\) between \({8.3 - 2.12 = 6.18}\) and \({8.3 + 2.12 = 10.42}\) and reject outside.

Step 4. Four outcomes. The truth is \(H_0\) or not; our decision is reject or keep. Two of the four combinations are mistakes.

A two by two table of boxes. Columns: H0 is true, H0 is false, under the note the truth, which we never see. Rows: reject H0 and keep H0, under the note our decision. Reject with H0 true: a reddish-purple box Type I error, rate alpha, chosen. Reject with H0 false: a green box correct, rate 1 minus beta, the power. Keep with H0 true: a green box correct, rate 1 minus alpha. Keep with H0 false: a reddish-purple box Type II error, rate beta.
S02.5.3-F2

The four outcomes of a test. A Type I error rejects a true \(H_0\); a Type II error keeps a false one. We choose \(\alpha\); \(\beta\) follows from the design and the truth.

The court analogy helps. \(H_0\) is “innocent”. Convicting an innocent person is a Type I error; acquitting a guilty one is a Type II error. A court that demands proof beyond reasonable doubt chooses a small \(\alpha\), and pays for it with more guilty people walking free.

Step 5. The same picture in machine learning. A classifier scores each email and flags it as spam when the score passes a threshold \(c\) (ML01.6.4). The scores of genuine emails and of spam are two overlapping distributions, exactly like the two curves above.

Two bell curves of model scores: a black one on the left labelled scores of negatives (like H0) and a sky-blue one on the right labelled scores of positives (like H1). An orange vertical line is labelled threshold c (the critical value). The part of the black curve beyond the threshold is solid reddish purple, labelled false-positive rate (like alpha); the part of the blue curve below the threshold is hatched reddish purple, labelled false-negative rate (like beta).
S02.5.3-F6

A classifier’s threshold is a critical value. Negatives scored above it are false positives (the Type I area); positives scored below it are false negatives (the Type II area).

The false-positive rate is \(\alpha\) and the false-negative rate is \(\beta\). Raising the threshold cuts false alarms and misses more spam. The ROC curve of ML01.7.3 draws every such trade at once.

The formal statement

NoteDefinitions

A test at level \(\alpha\) rejects \(H_0\) when the p-value is at most \(\alpha\).

  • Type I error: rejecting \(H_0\) when it is true. \({\Prob(\text{reject} \mid H_0) = \alpha}\).
  • Type II error: keeping \(H_0\) when it is false. Its rate is \(\beta\), which depends on which alternative is true.
  • Power: \({1 - \beta}\), the chance of rejecting a false \(H_0\) (the subject of S02.6).

Why the Type I rate is exactly \(\alpha\). Under \(H_0\) the statistic \({Z}\) follows \({N(0, 1)}\), so for the one-sided test

\[ \begin{aligned} \Prob(\text{reject} \mid H_0) &= \Prob(Z \ge z_{1-\alpha}) \\ &= 1 - \Phi(z_{1-\alpha}) \\ &= 1 - (1 - \alpha) = \alpha . \end{aligned} \]

The same follows from the last lesson: a p-value that is uniform under \(H_0\) is at most \(\alpha\) with chance \(\alpha\). For a count the p-value is only “at most uniform”, so the Type I rate is at most \(\alpha\).

The Type II rate for one alternative. Suppose the truth is \({\mu = \mu_1}\) with \({\mu_1 > \mu_0}\). Then \({\bar{x} \sim N(\mu_1, \mathrm{SE}^2)}\), where \({\mathrm{SE} = \sigma/\sqrt{n}}\). We keep \(H_0\) when \({\bar{x} < c}\), so

\[ \begin{aligned} \beta &= \Prob(\bar{x} < c \mid \mu = \mu_1) \\ &= \Phi\Big(\frac{c - \mu_1}{\mathrm{SE}}\Big) \\ &= \Phi\Big(z_{1-\alpha} - \frac{\mu_1 - \mu_0}{\mathrm{SE}}\Big). \end{aligned} \]

The second line standardises \(\bar{x}\) with the alternative’s mean. The third substitutes \({c = \mu_0 + z_{1-\alpha}\,\mathrm{SE}}\) and splits the fraction. The formula shows every lever at once: a farther alternative, a smaller \(\mathrm{SE}\) (more data) or a larger \(\alpha\) each make \(\beta\) smaller. S02.6 builds on it.

The trade. Fix the data and move \(c\). The Type I rate \({1 - \Phi((c - \mu_0)/\mathrm{SE})}\) falls as \(c\) rises; the Type II rate \({\Phi((c - \mu_1)/\mathrm{SE})}\) rises. One can be bought only with the other.

A plot of error rates from 0 to 1 against the critical value c from 7 to 12. A solid reddish-purple curve, the Type I rate alpha, falls from near 0.9 to near 0. A dashed reddish-purple curve, the Type II rate beta, rises from near 0 to near 0.9. They cross near c = 9.65. Orange dotted vertical lines mark c = 9.2, 10.08 and 11, with orange dots on both curves, labelled alpha 0.20, beta 0.05; alpha 0.05, beta 0.20; and alpha 0.006, beta 0.50.
S02.5.3-F4

Moving the critical value for the listing site, with the alternative \({\mu_1 = 11}\). At \({c = 9.2}\), \({\alpha = 0.20}\) and \({\beta = 0.05}\); at 10.08, 0.05 and 0.20; at 11, 0.006 and 0.50.

Go deeper: why fix alpha and not beta

The two hypotheses are not treated alike. \(H_0\) is a single precise value, so the Type I rate is a single number we can set. The alternative is usually a range (\({\mu > 8.3}\)), and \(\beta\) is a different number for each value in it: 0.84 if the truth is 9, 0.20 if it is 11, almost 0 if it is 14. So the convention is to cap the error we can compute exactly, the false alarm, and to study \(\beta\) as a curve over the alternatives. That curve, read as power, is the subject of S02.6.

Worked example

At level 0.05, is the listing site dearer than the city? And how often would this test miss a site whose true mean is 11?

A trace in two rows. Row 1, fix alpha before the data, the critical value: an orange box alpha = 0.05, so z 0.95 = 1.645, then an orange box c = 8.3 + 1.645 times 1.082 = 10.08. An arrow leads down to row 2, under H1 (mu = 11), the chance of keeping H0: a sky-blue box z = (10.08 minus 11)/1.082 = minus 0.85, then a reddish-purple box beta = Phi(minus 0.85) = 0.20. A note reads reject when x-bar is above 10.08; power = 1 minus 0.20 = 0.80.
S02.5.3-F3

The critical value comes from \(\alpha\) and \(H_0\) alone; \(\beta\) needs a specific alternative.

Level and region. \({\alpha = 0.05}\), one-sided, so \({z_{0.95} = 1.645}\) and \({c}\) \({= 8.3 + 1.645 \times 1.082}\) \({= 8.3 + 1.780}\) \({= 10.08}\). Reject when \({\bar{x} > 10.08}\).

Decision. The site’s \({\bar{x} = 10.5}\) is above 10.08, so we reject \(H_0\). The p-value route agrees: \({0.021 \le 0.05}\). Had we chosen \({\alpha = 0.01}\) the critical value would be 10.82 and we would keep \(H_0\) (exercise 1). The data are the same; the decision depends on a level fixed in advance, which is why it must be fixed in advance.

Type II rate at \({\mu_1 = 11}\). Standardise the critical value with the alternative’s mean: \({(10.08 - 11)/1.082 = -0.85}\), so \({\beta = \Phi(-0.85) = 0.20}\). A site whose flats really average 11 would slip through this test one time in five; the power against 11 is 0.80.

Type II rate at \({\mu_1 = 9}\). \({(10.08 - 9)/1.082 = 1.00}\), so \({\beta = \Phi(1.00) = 0.84}\). A site that is dearer by 0.7 would be missed 84 percent of the time with 25 flats.

Code

From scratch, the quantile \({z_{1-\alpha}}\) is found by bisection on \(\Phi\): halve an interval that contains it until it is tiny.

import math

def Phi(z):
    return 0.5 * (1 + math.erf(z / math.sqrt(2)))

def Phi_inv(q, lo=-10.0, hi=10.0):
    """The z with Phi(z) = q, by bisection: Phi is increasing, so keep the half that contains q."""
    for _ in range(100):
        mid = (lo + hi) / 2
        lo, hi = (mid, hi) if Phi(mid) < q else (lo, mid)
    return (lo + hi) / 2

mu0, sigma, n = 8.3, 5.41, 25
se = sigma / math.sqrt(n)
z95 = Phi_inv(0.95)
c = mu0 + z95 * se
alpha = 1 - Phi((c - mu0) / se)
beta11 = Phi((c - 11) / se)
beta9 = Phi((c - 9) / se)
lo2, hi2 = mu0 - Phi_inv(0.975) * se, mu0 + Phi_inv(0.975) * se
print(f"z_0.95 = {z95:.3f}, critical value c = {c:.2f}")
print(f"alpha = {alpha:.3f}, beta at 11 = {beta11:.3f}, beta at 9 = {beta9:.3f}")
print(f"two-sided: keep H0 for x-bar from {lo2:.2f} to {hi2:.2f}")
z_0.95 = 1.645, critical value c = 10.08
alpha = 0.050, beta at 11 = 0.198, beta at 9 = 0.841
two-sided: keep H0 for x-bar from 6.18 to 10.42

Now a simulation of both error rates: 20,000 sites of 25 normal rents where \(H_0\) is true, and 20,000 where the mean is 11. Count how often the rule rejects.

import numpy as np

rng = np.random.default_rng(3)
null_means = rng.normal(mu0, sigma, size=(20_000, n)).mean(axis=1)
alt_means = rng.normal(11.0, sigma, size=(20_000, n)).mean(axis=1)
type1 = np.mean(null_means > c)
type2 = np.mean(alt_means <= c)
print(f"Type I rate {type1:.4f} (promised 0.05); Type II rate {type2:.4f} (formula {beta11:.4f})")
Type I rate 0.0505 (promised 0.05); Type II rate 0.1969 (formula 0.1975)

Finally the library: norm.ppf is \({\Phi^{-1}}\) and norm.sf is \({1 - \Phi}\). loc and scale shift and stretch the standard normal, so the critical value and \(\beta\) can be read off directly on the scale of \(\bar{x}\).

from scipy import stats

c_lib = stats.norm.ppf(0.95, loc=mu0, scale=se)
print(f"critical value {c_lib:.4f}")
print(f"alpha {stats.norm.sf(c_lib, loc=mu0, scale=se):.4f}")
print(f"beta at 11 {stats.norm.cdf(c_lib, loc=11, scale=se):.4f}")
critical value 10.0797
alpha 0.0500
beta at 11 0.1975

Explore

This module’s interactive, the p-value machine, is in lesson S02.5.2: run a thousand fair-coin experiments and read off the share of p-values at or below 0.05, which is the Type I error rate of a level-0.05 test. The overlap diagram above becomes a live explorer in the next module, S02.6, where sliders move the two curves and the critical value.

Pitfalls

Two panels. Left, marked with a cross and titled Kept, so mu = 8.3: a black curve at 8.3 and a sky-blue curve at 9 overlap heavily, with the orange critical line at 10.08 and most of the blue curve hatched reddish purple; text says if in truth mu = 9, the test keeps H0 84 percent of the time. Right, marked with a tick and titled Kept: 8.3 is not ruled out: a vermillion bar with a dot at 9.2 spans 7.08 to 11.32 over a number line, labelled 95 percent interval from x-bar = 9.2: 7.08 to 11.32, with black triangles at 8.3 and 11 both under the bar; text says mu = 8.3 and mu = 11 both lie inside the interval: the data rule out neither.
S02.5.3-F5

Wrong beside right. A sample mean of 9.2 keeps \(H_0\), but a test with 25 flats keeps \(H_0\) 84 percent of the time even when the truth is 9. Kept means “not ruled out”, and the interval shows that 11 is not ruled out either.

“We kept \(H_0\), so the site is typical.”

Keeping \(H_0\) means the data are compatible with it, not that it is true. With 25 flats the test would keep \(H_0\) 84 percent of the time even if the site’s mean were 9. The court does not declare the defendant innocent; it says “not proven”. Report what the data allow, for example with the interval of S02.4, as the right panel does and as S02.5.4 explains.

“At level 0.05, a rejection has a 5 percent chance of being wrong.”

\(\alpha\) is \({\Prob(\text{reject} \mid H_0)}\), the false-alarm rate among true nulls. The share of rejections that are wrong is a different conditional probability; in the 1,000-test example of the last lesson it was 47 percent.

“The p-value came out at 0.07, so this time I’ll use \(\alpha = 0.1\).”

Choosing \(\alpha\) after seeing the data removes the guarantee that made it worth choosing. The same mistake in machine learning is picking a classifier’s threshold on the test set: the reported error rates are then too good, because the threshold was fitted to the very examples that score it.

Check yourself

Recall

1. What is a Type I error, and what is its probability in a level-\(\alpha\) test?

Show answer

Rejecting \(H_0\) when it is true. Its probability is \(\alpha\) (at most \(\alpha\) for a count).

2. What is a Type II error, and why is its rate not a single number?

Show answer

Keeping \(H_0\) when it is false. Its rate \(\beta\) depends on which alternative value is true.

3. Write the critical value of the one-sided z-test on the scale of \(\bar{x}\), and its value for the listing site at \({\alpha = 0.05}\).

Show answer

\({c}\) \({= \mu_0 + z_{1-\alpha}\,\sigma/\sqrt{n}}\); \({8.3 + 1.645 \times 1.082 = 10.08}\).

4. What happens to \(\alpha\) and \(\beta\) when the critical value moves up, with the data fixed?

Show answer

\(\alpha\) falls and \(\beta\) rises: one error is traded for the other.

5. In a classifier, which rates play the parts of \(\alpha\) and \(\beta\)?

Show answer

The false-positive rate (negatives above the threshold) is \(\alpha\); the false-negative rate (positives below it) is \(\beta\).

Exercises

1. ★ At \({\alpha = 0.01}\) (one-sided, \({z_{0.99} = 2.326}\)), find the critical value for the listing site. Is \({\bar{x} = 10.5}\) rejected?

Show solution

\({c}\) \({= 8.3 + 2.326 \times 1.082}\) \({= 8.3 + 2.517}\) \({= 10.82}\). Since \({10.5 < 10.82}\) we keep \(H_0\). The p-value 0.021 agrees: it is above 0.01.

2. ★★ With \({\alpha = 0.01}\), what is the Type II rate against \({\mu_1 = 11}\)? Compare it with the 0.20 at \({\alpha = 0.05}\).

Show solution

\({(\mu_1 - \mu_0)/\mathrm{SE}}\) \({= 2.7/1.082}\) \({= 2.495}\), so

\[ \begin{aligned} \beta &= \Phi(2.326 - 2.495) \\ &= \Phi(-0.169) = 0.43 . \end{aligned} \]

Cutting the false-alarm rate from 0.05 to 0.01 more than doubles the chance of missing a site whose mean is 11, from 0.20 to 0.43.

3. ★★★ Keep \({\alpha = 0.05}\) and the alternative \({\mu_1 = 11}\), but sample 100 flats instead of 25. Find the new critical value and \(\beta\), and explain in terms of the overlap diagram why both errors can now be small at once.

Show solution

\({\mathrm{SE} = 5.41/\sqrt{100} = 0.541}\). The critical value is \({c = 8.3 + 1.645 \times 0.541 = 9.19}\). Then

\[ \begin{aligned} \beta &= \Phi\Big(1.645 - \frac{2.7}{0.541}\Big) \\ &= \Phi(1.645 - 4.991) \\ &= \Phi(-3.35) = 0.0004 . \end{aligned} \]

Four times the data halves the standard error, so both curves become half as wide while their centres stay at 8.3 and 11. They hardly overlap any more, and a single line can leave very little of either curve on the wrong side. The trade between \(\alpha\) and \(\beta\) is forced only for a fixed amount of data.

Summary card

Fix \(\alpha\) first; then two curves and one line show both ways to be wrong.

The anchor figure again: the null curve at 8.3 and the alternative curve at 11, the orange critical value at 10.08, the solid Type I area alpha = 0.05 under the null curve and the hatched Type II area beta = 0.20 under the alternative.
S02.5.3-F1

Solid: Type I. Hatched: Type II.

  • Decision. Reject \(H_0\) when the p-value is at most \(\alpha\), fixed before the data.
  • Region. One-sided \({z \ge z_{1-\alpha}}\); two-sided \({\abs{z} \ge z_{1-\alpha/2}}\).
  • Errors. Type I rate \(\alpha\); Type II rate \({\beta}\) \({= \Phi(z_{1-\alpha} - (\mu_1 - \mu_0)/\mathrm{SE})}\).
  • Listing site. \({c = 10.08}\); \({\beta = 0.20}\) against 11; \({\beta = 0.84}\) against 9.
  • Trade. Moving the line trades \(\alpha\) for \(\beta\), like a classifier’s threshold; more data shrinks both.

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • Jeremy Orloff and Jonathan Bloom, Reading 17b, Null Hypothesis Significance Testing I, MIT OpenCourseWare 18.05 (Spring 2022). Section 4.2 tabulates the two types of error, Section 4.3 defines the significance level as P(reject H0 | H0) and power as P(reject H0 | HA) = 1 minus P(type II error), and Section 7 states the p-test: reject when the p-value is less than the significance level.
  • David Diez, Mine Cetinkaya-Rundel and Christopher Barr, OpenIntro Statistics, fourth edition (free PDF). Section 5.3.3, decision errors, with the court analogy for Type 1 and Type 2 errors, and Section 5.3.5, choosing a smaller or larger significance level according to the cost of each error.
  • SciPy developers, scipy.stats.norm: the methods ppf, cdf and sf, with loc and scale for a normal of any mean and spread.

Next. S02.5.4 Tests and intervals, significance and importance shows that a two-sided test and a confidence interval are one object, and separates a significant result from an important one.

This lesson unlocks

S02.5.4 Tests and intervals, significance and importance S02.6 Power, effect size and sample size

Your progress is saved in this browser only. No account, no tracking.
Back to top