\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

S01.8 · An analysis workflow from start to finish

A free, illustrated module of 4 lessons and one interactive in S01 Exploratory Data Analysis: A question, a table and a plan; Audit before you analyse; From single columns to relationships; Report what you found and what you cannot claim.

S01.8 L1 · Core 4 lessons · about 100 min Live · in review

Part of S01 Exploratory Data Analysis in the Statistics and Data Analysis track.

The earlier modules each taught one tool. This one uses them together, in order, on a real table: 442 diabetes patients, ten measurements from a first visit and a score of how far the disease had progressed a year later. Start from a question and the data’s history, audit the table before trusting a number, look at one column at a time and then at pairs and subgroups, and finish with a one-page report that says what was found, how much was looked at, and what the data cannot claim.

What you need first. The whole of S01: data types and measurement (S01.1), sampling and bias (S01.2), centre, spread and shape (S01.3), distributions and outliers (S01.4), relationships (S01.5), missing data (S01.6) and Simpson’s paradox (S01.7), with the pandas of P02.

Lessons

1. A question, a table and a plan S01.8.1 25 min

   An analysis starts before the first mean: with a question, with where the data came from, and with a dictionary that gives every column a meaning, a unit and a scale. Meet the seven-step workflow and the capstone table of 442 diabetes patients.

2. Audit before you analyse S01.8.2 25 min

   Check that every column is what its dictionary says before computing a single summary: types, ranges, distinct values, missing values and their stand-ins, duplicated rows. Write the checks as one audit function and as assertions, and keep a log of every fix.

3. From single columns to relationships S01.8.3 25 min

   Look at each column alone, then at pairs, then inside subgroups. On the diabetes table: a right-skewed progression score, body mass index leading at r = 0.59, a near-duplicate pair explained by a formula, and a relationship that holds in every group.

4. Report what you found and what you cannot claim S01.8.4 25 min

   Turn an exploration into one honest page: the question, the data, what was checked, three to five findings each with one number and one picture, the limits, and the next step. Count how much was looked at, and make every number rebuild from the raw table.

The interactive

S01.8-X1 · Audit trail. A copy of the flats table with five planted faults. Run the workflow’s checks one at a time to light up the cells each one catches, then fix or ignore each fault and watch the mean and median rent and the correlation of area with rent move back towards the clean table’s values. It lives in Audit before you analyse.