\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

P01.10 · Iterators, generators and comprehensions

A free, illustrated module of 4 lessons and one interactive in P01 Python from Zero: Comprehensions; Iterables and iterators; Generators; Lazy pipelines and functions as arguments.

P01.10 L0 · Foundation 4 lessons · about 100 min Live · in review

Part of P01 Python from Zero in the Programming and Data track.

A for loop hides a small machine: it asks for an iterator and calls next until the items run out. These four lessons write the loop-and-append pattern as one comprehension, open up the for loop to show iter, next and StopIteration, build iterators of your own as classes and then as generator functions that pause at each yield, and chain map, filter and islice into lazy pipelines that read a file one row at a time, the way data too large for memory is read.

What you need first. Loops and loop patterns (P01.3), functions and the call stack (P01.4), lists, tuples, dictionaries and sets with zip and enumerate (P01.6), reading text files (P01.7), try and except (P01.8.2), and classes with special methods (P01.9).

Lessons

1. Comprehensions P01.10.1 25 min

   The loop that makes an empty list and appends to it, written as one expression. Build a list comprehension part by part (output expression, for clause, if clause), then dictionary and set comprehensions, and learn when a plain loop is clearer.

2. Iterables and iterators P01.10.2 25 min

   What a for loop really does: it asks for an iterator with iter, calls next until StopIteration, and stops. Write the loop by hand, build a Countdown iterator class, and see why a zip or a file can be looped over only once.

3. Generators P01.10.3 25 min

   A function that contains yield returns a generator: its frame pauses at each yield and resumes at the next call to next. Countdown in four lines, generator expressions as lazy comprehensions, and batches that cut the study table into the mini-batches of machine learning.

4. Lazy pipelines and functions as arguments P01.10.4 25 min

   A function is an object, so it can be handed to another function: sorted and max with key=, lambda, and map and filter as lazy stages, with islice to stop early. Ends with study.csv read as one pipeline, one row in memory at a time, to the mean of the passing scores, 65.0.

The interactive

P01.10-X1 · Lazy and eager pipeline. Ten numbered items pass through a filter and a map to a consumer that keeps the first few. In eager mode each stage builds its whole list before the next starts; in lazy mode each step pulls one item through every stage. Read-outs count the calls made, the items held in lists and the items read, and show the lazy pipeline stopping early. It lives in Generators.