\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

First tests

A test is code that checks code against a value worked out by hand. Write tests for mean and median, run them with a runner built from scratch and then with pytest, compare floats with pytest.approx, and pin a bug so that it cannot come back.

P01.8.4L0 Foundation25 minIn review

Before this lesson

P01.8.3 Debugging as a method P01.5.3 Installing packages with pip

After this lesson you can

• Write a test as a function that runs code on a small input and uses assert to compare the result with a value worked out by hand.
• Choose the cases for a function: a usual case, the smallest case, a case for each branch, and an error case.
• Run tests with a runner written from scratch and with pytest, and read pytest’s report.
• Compare floats with pytest.approx and check that an error is raised with pytest.raises.
• Pin a bug with a test written before the fix, so that the bug cannot come back unnoticed.

In review. The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

The question

In P01.8.3 a median gave 65 instead of 60.0, and nothing on the screen said so. You found it by checking cases by hand. Next week you will change stats_tools.py again, and you will not check every case again by hand. How can the computer recheck every function, every time, against numbers you trust?

From the last lesson

1. Why must you predict a value before you look at it?

Show answer

Without a prediction a printed value cannot be called right or wrong, so it rules nothing out.

2. How many looks does halving need to find one bad line among 1,000?

Show answer

Ten: \({2^{10} = 1024}\) is the first power of 2 that reaches 1,000.

Intuition

A test is a few lines of code that run your code on a small input and compare the result with the answer you worked out by hand. If the two agree, the test passes and says nothing. If not, it fails loudly, with the input, the result and the expected value.

P01.8.4-F1

A test runs code on a small input and compares the result with a value worked out by hand. The same test passes on the repaired median and fails on the one with the bug.

The picture holds three ideas. The input is small, so the answer can be worked out by hand. The expected value comes from the hand calculation, never from the code being tested. And the test is code, so it can be run again after every change, in a second.

Build it up

Step 1. The assert statement. assert is followed by something that should be true. If it is true, nothing happens. If it is false, Python raises AssertionError.

from stats_tools import mean, median

assert median([55, 65, 80, 40, 60]) == 60      # true: nothing happens
print('the first assert said nothing')

try:
    assert median([55, 65, 80, 40]) == 65      # false: the median is 60.0
except AssertionError:
    print('the second assert raised AssertionError')

the first assert said nothing
the second assert raised AssertionError

Step 2. A test is a function. Wrap an assert in a function whose name starts with test_ and says what it checks. Calling it runs the check.

def test_median_of_an_even_count():
    assert median([55, 65, 80, 40]) == 60.0


test_median_of_an_even_count()
print('passed')

passed

Step 3. Choose the cases. One case proves little, as P01.8.3 found. Pick cases of every kind: a usual one, the smallest one, one for each branch of the code, and one where an error is the right answer.

P01.8.4-F2

Four cases chosen so that each branch of median has a case of its own. Together they run every line.

The four cases for median are five scores (60), one score (55), four scores (60.0) and no scores at all, where the right answer is a ValueError. MIT’s 6.100L calls choosing cases from the paths through the code glass-box testing, and adds the boundary cases worth trying: an empty list, a list of one.

Step 4. Run them all. A test runner calls every test and counts the passes and failures. The worked example writes one from scratch; then pytest does the same job for you.

The formal statement

NoteDefinitions: test, assertion, regression test

A test is a function that runs code on a chosen input and checks the result with one or more assertions. The statement assert condition does nothing when the condition is true and raises AssertionError when it is false; assert condition, message adds a message. A regression test pins a bug that was once found, so that it fails if the bug ever comes back.

The Python Language Reference (7.3) defines assert condition as the same as

if __debug__:
    if not condition:
        raise AssertionError

where __debug__ is True unless Python was started with the option -O, which removes every assert. That is why asserts belong in tests, and why a function checks its own input with if and raise instead (P01.8.2).

pytest. pytest is a package installed with pip (P01.5.3), here at version 9.2.0, the version this site pins. Run in a folder with python -m pytest, it

1. finds every file named test_*.py or *_test.py in the folder and below,
2. runs every function in them whose name starts with test,
3. prints one mark per test, . for a pass and F for a failure, then a report on each failure, and the totals.

Two helpers cover what a plain == cannot.

• pytest.approx(expected) compares floats with a tolerance: by default a result is equal if it is within one part in a million of expected, or within \({10^{-12}}\) of it, which matters only near zero.
• with pytest.raises(ValueError): passes only if the block inside it raises a ValueError (or one of its kinds), and fails if it raises nothing.

Worked example

Here are eight tests, four for each function: a usual case, one value, an even count, and the empty list. The float test uses the tolerance of P01.2.4, and the error tests catch the ValueError they expect with try and except.

def test_mean_of_the_scores():
    assert mean([55, 65, 80, 40, 60]) == 60.0


def test_mean_of_one_score():
    assert mean([55]) == 55.0


def test_mean_of_two_floats():
    assert abs(mean([0.1, 0.2]) - 0.15) < 1e-9


def test_mean_of_no_scores_is_an_error():
    try:
        mean([])
    except ValueError:
        return
    assert False, 'mean([]) raised nothing'


def test_median_of_the_scores():
    assert median([55, 65, 80, 40, 60]) == 60


def test_median_of_one_score():
    assert median([55]) == 55


def test_median_of_an_even_count():
    assert median([55, 65, 80, 40]) == 60.0


def test_median_of_no_scores_is_an_error():
    try:
        median([])
    except ValueError:
        return
    assert False, 'median([]) raised nothing'

The runner. A function is a value like any other (P01.10.4 makes much of this), so a list can hold the tests, and test() calls each one in turn; test.__name__ is the name it was defined with.

def run_tests(tests):
    passed = 0
    failed = 0
    for test in tests:
        try:
            test()
        except AssertionError:
            print('FAIL', test.__name__)
            failed += 1
        else:
            passed += 1
    print(passed, 'passed,', failed, 'failed')


TESTS = [test_mean_of_the_scores, test_mean_of_one_score, test_mean_of_two_floats,
         test_mean_of_no_scores_is_an_error, test_median_of_the_scores, test_median_of_one_score,
         test_median_of_an_even_count, test_median_of_no_scores_is_an_error]
run_tests(TESTS)

8 passed, 0 failed

Now put the second bug of P01.8.3 back, as a careless edit might, and run the same tests. The tests look up the name median when they run (P01.4.4), so they test the new one.

def median(values):          # the upper middle again, for an even count
    if len(values) == 0:
        raise ValueError('median of an empty list')
    ordered = sorted(values)
    return ordered[len(ordered) // 2]


run_tests(TESTS)
from stats_tools import median          # the repaired median again

FAIL test_median_of_an_even_count
7 passed, 1 failed

P01.8.4-F3

The runner’s trace: eight tests against two versions of median. Only the test written for the second bug catches it, and that is its whole job.

The test test_median_of_an_even_count is a regression test. It was written for the bug of P01.8.3, worked out by hand as \({(55 + 65) / 2} = 60.0\), and it fails whenever that bug is in the code. Written before the fix, it fails first, which proves that it can catch the bug; after the fix it passes, and it stays in the file to guard the fix. MIT’s 6.100L describes the habit: “add test for bugs as you find them”.

Code

The library version: the same tests in a file of their own, for pytest. pytest.approx and pytest.raises replace the hand-made tolerance and the try blocks.

test_stats_tools.py

import pytest

from stats_tools import mean, median


def test_mean_of_the_scores():
    assert mean([55, 65, 80, 40, 60]) == 60.0


def test_mean_of_one_score():
    assert mean([55]) == 55.0


def test_mean_of_two_floats():
    assert mean([0.1, 0.2]) == pytest.approx(0.15)


def test_mean_of_no_scores_is_an_error():
    with pytest.raises(ValueError):
        mean([])


def test_median_of_the_scores():
    assert median([55, 65, 80, 40, 60]) == 60


def test_median_of_one_score():
    assert median([55]) == 55


def test_median_of_an_even_count():
    assert median([55, 65, 80, 40]) == 60.0


def test_median_of_no_scores_is_an_error():
    with pytest.raises(ValueError):
        median([])

Put it beside stats_tools.py and run pytest in that folder, with -q for a short report:

python -m pytest -q

Here is the report with the second bug put back in stats_tools.py. (This page runs that command on the two files in a temporary folder, in a terminal 52 characters wide, and cuts the time from the last line.)

......F.                                     [100%]
===================== FAILURES =====================
___________ test_median_of_an_even_count ___________

    def test_median_of_an_even_count():
>       assert median([55, 65, 80, 40]) == 60.0
E       assert 65 == 60.0
E        +  where 65 = median([55, 65, 80, 40])

test_stats_tools.py:32: AssertionError
============= short test summary info ==============
FAILED test_stats_tools.py::test_median_of_an_even_count
1 failed, 7 passed

P01.8.4-F5

pytest’s report, read part by part. Like a traceback, it is read from the bottom: the totals, the summary, then the failure itself.

Repair median and run pytest again:

........                                     [100%]
8 passed

The two helpers, tried on their own:

import pytest

print(0.1 + 0.2 == 0.3, 0.1 + 0.2 == pytest.approx(0.3))
print(mean([0.1, 0.2]), mean([0.1, 0.2]) == pytest.approx(0.15))
with pytest.raises(ValueError):
    median([])
print('median([]) raised ValueError, as the test expects')

False True
0.15000000000000002 True
median([]) raised ValueError, as the test expects

Explore

This module’s interactive, the traceback explorer, lives in P01.8.1. Read its program as a tester would: for each of its four inputs, write down by hand what the program should print before you press Step. Each input is a test case; the three that fail are the cases a test with pytest.raises would pin.

Pitfalls

P01.8.4-F4

Wrong beside right. An expected value copied from the code’s own output turns the bug into the answer. An expected value worked out by hand catches it.

“I ran the code, saw 65, and put 65 in the test.”

Then the test checks that the code does what it does, bugs included, and it will fail the day the bug is fixed. Work the expected value out by hand on a case small enough to do so. This is the whole reason tests use tiny inputs.

“assert mean([0.1, 0.2]) == 0.15 must pass.”

mean([0.1, 0.2]) is 0.15000000000000002 (P01.2.1), so == is False. Compare floats with a tolerance: pytest.approx(0.15), or abs(a - b) < 1e-9 by hand.

“An assert inside my function checks its input well enough.”

Run with python -O, Python removes every assert, and the check is gone. Check input with if and raise ValueError(...) (P01.8.2); keep assert for tests.

“My test passed, so the code is right.”

A test can pass for the wrong reason: a function named check_mean instead of test_mean is never collected by pytest, and a test without an assert can never fail. Watch each new test fail once, on purpose or on the bug it was written for, before you trust it. In machine learning this is the main defence: as Karpathy puts it, neural net training “fails silently”, so small cases worked by hand are how numerical code is trusted.

Check yourself

Recall

1. What does assert condition do when the condition is true, and when it is false?

Show answer

When it is true, nothing. When it is false, it raises AssertionError.

2. Which files and which functions does pytest run?

Show answer

Files named test_*.py or *_test.py in the folder and below it, and in them the functions whose names start with test.

3. Why should the expected value of a test be worked out by hand?

Show answer

If it is copied from the code’s own output, the test checks the code against itself and passes on the bug.

4. What do pytest.approx and pytest.raises check?

Show answer

pytest.approx(x) makes == true when a float is within a small tolerance of x (by default one part in a million). pytest.raises(E) passes only if the block inside it raises an exception of type E or one of its kinds.

5. What is a regression test?

Show answer

A test written for a bug that was found, so that it fails if the bug ever comes back, such as test_median_of_an_even_count for the upper-middle bug.

Exercises

1. ★ stats_tools.py also holds predict(hours, slope=9, intercept=33). Write two tests for it, working the expected values out by hand: one for 2 hours with the default line, and one for 2 hours with slope 10.

Show solution

By hand, \({9 \times 2 + 33} = 51\) and \({10 \times 2 + 33} = 53\).

from stats_tools import predict


def test_predict_two_hours():
    assert predict(2) == 51


def test_predict_two_hours_with_slope_10():
    assert predict(2, slope=10) == 53

2. ★★ A test reads assert mean([0.1, 0.1, 0.1]) == 0.1. Does it pass? Fix it in two ways: once without pytest, once with it.

Show solution

sum([0.1, 0.1, 0.1]) is 0.30000000000000004, and dividing by 3 gives 0.10000000000000002, so == is False and the test fails although the code is right. Without pytest: assert abs(mean([0.1, 0.1, 0.1]) - 0.1) < 1e-9. With pytest: assert mean([0.1, 0.1, 0.1]) == pytest.approx(0.1).

3. ★★★ The runner of the worked example catches only AssertionError. What happens if a test raises another exception, for example an old median without the guard, where median([]) raises IndexError? Change run_tests so that such a test is counted as failed, with the type of the error printed, and the runner carries on.

Show solution

The IndexError is not caught by except AssertionError, so it leaves run_tests and stops the whole run with a traceback; the tests after it never run. A runner is one of the rare places where catching every ordinary exception is right, because its job is to report each one, not to hide it.

def run_tests(tests):
    passed = 0
    failed = 0
    for test in tests:
        try:
            test()
        except AssertionError:
            print('FAIL ', test.__name__)
            failed += 1
        except Exception as error:
            print('ERROR', test.__name__, type(error).__name__, error)
            failed += 1
        else:
            passed += 1
    print(passed, 'passed,', failed, 'failed')

With the unguarded median, test_median_of_no_scores_is_an_error now prints ERROR test_median_of_no_scores_is_an_error IndexError list index out of range, and the other seven tests still run. pytest does the same: an unexpected exception in a test is reported as a failure, with its traceback.

Summary card

A test runs code on a tiny input and compares the result with a value worked out by hand.

P01.8.4-F1

Expected values come from your hand, not the code.

• A test: def test_...(): with assert result == expected.
• The cases: a usual one, the smallest, one per branch, and an error case.
• pytest: finds test_*.py, runs every test... function, prints . or F and a report.
• Helpers: == pytest.approx(x) for floats; with pytest.raises(E): for errors.
• Regression test: pin each bug you find, so that it cannot come back.

Sources and next steps

Sources. Each was opened and checked against this lesson on 12 November 2026.

• pytest developers, Get Started (version 9.2.0): a test function with a plain assert, that pytest runs “all files of the form test_.py or _test.py”, that [100%] is the overall progress, pytest.raises, and pytest.approx for floats; and the API reference: approx uses a relative tolerance of 1e-6 and an absolute tolerance of 1e-12, and raises asserts that a block raises “an exception type, or one of its subclasses”. The rule for collecting test-prefixed functions is in Good Integration Practices.
• Python Software Foundation, The Python Language Reference, 7.3: The assert statement: the equivalent if __debug__: code, and that no code is emitted for an assert when optimisation is requested with -O.
• Ana Bell, MIT OpenCourseWare 6.100L, Fall 2022. Lecture 12: unit and regression testing (“add test for bugs as you find them”), black-box and glass-box testing, and boundary cases such as empty and single-item lists. Lecture 13: assertions as a defensive programming tool.
• Andrej Karpathy, A Recipe for Training Neural Networks, 25 April 2019: the second of its opening observations, “Neural net training fails silently”.

Next. Module P01.9 begins with classes: new types of your own, each holding its own data, the form in which a fitted model keeps what it has learned. Testing grows into a habit of whole projects in P06.4.

This lesson unlocks

P01.9.1 Classes and instances P06.4 Testing and continuous integration (planned)

Your progress is saved in this browser only. No account, no tracking.