\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML01.10 · A full project from raw data to report

A free, illustrated module of 4 lessons and one interactive in ML01 Machine Learning Foundations: Frame the problem and set a baseline; Explore and prepare the data; Train, tune and compare; Analyse errors and write it up.

ML01.10 L1 · Core 4 lessons · about 100 min Live · in review

Part of ML01 Machine Learning Foundations in the Machine Learning track.

The earlier modules each taught one tool. This one uses them together, in order, on one real table: 442 diabetes patients and a score for how far the disease had progressed a year later. The four lessons are the four stages of a project. Frame the question, choose a metric from the cost of errors and set a baseline. Look at the data and prepare it inside a pipeline, so that nothing leaks. Train, tune and compare three kinds of model with honest uncertainty. Then find where the chosen model is wrong, open the test set once, and write a report that says what the model can and cannot be trusted to do.

What you need first. Train, validation and test sets and leakage (ML01.2), k-nearest neighbours and standardising features (ML01.3), linear regression with many features (ML01.4), regression metrics (ML01.7), and the lasso and cross-validation (ML01.9). The standard error of a mean (ML01.2.3) is used throughout.

Lessons

1. Frame the problem and set a baseline ML01.10.1 25 min

   Before any model is trained, a project needs a frame: what is predicted, from what, how an error is scored and which score a model must beat. This lesson frames a real medical dataset, chooses the metric from the cost of errors and sets the baseline.

2. Explore and prepare the data ML01.10.2 25 min

   Look before you model: the range of every column, the gaps, the shape of the target and what moves with it. Then prepare the features inside a pipeline, so that every number it learns comes from training rows only and nothing leaks from the rows held out.

3. Train, tune and compare ML01.10.3 25 min

   Three kinds of model, linear regression, k-nearest neighbours and the lasso, are tuned and scored on the same five folds. Their scores come with standard errors, the differences are compared fold by fold, and a rule fixed in advance picks the winner.

4. Analyse errors and write it up ML01.10.4 25 min

   An average error hides who the model fails. This lesson breaks the chosen model’s errors down by group, shows why errors lean against the outcome even for a sound model, opens the test set exactly once, puts an interval on the result and writes the one-page report.

The interactive

ML01.10-X1 · Project board. Build a pipeline stage by stage on the diabetes data: raw or standardised features, a linear model or k-nearest neighbours, a ridge or lasso penalty and its strength. The board shows the validation score of every setting against the baseline, and what happens to honesty when you open the test set more than once. It lives in Train, tune and compare.