\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

P07 · Data Engineering

A free, illustrated L2 to L3 course of 9 modules in the Programming and Data track. Modules include: Data lifecycle and architectures; File formats and storage; Batch pipelines and orchestration.

P07 L2 to L3 · Intermediate to Advanced 9 modules · about 36 lessons Planned · Ring 3

Part of the Programming and Data track. Ships in Ring 3 (Breadth) of the roadmap.

Modules

Data lifecycle and architectures
File formats and storage
Batch pipelines and orchestration
Warehouses, lakes and lakehouses
Distributed processing with Spark
Streaming and event systems
Data modeling and transformation
Data quality, lineage and governance
Cost, scale and reliability

Signature illustrations

The pictures a learner should still see in their head a year later:

Pipeline graphs, partition and shuffle diagrams, row versus columnar layout, windowing timelines for streams.

Status

This course is planned, not written yet. It ships with Ring 3 (Breadth).

Every lesson will follow the same twelve-part blueprint with at least three purpose-made figures. Watch the repository to hear when modules ship, or help write and review it.