\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

S01.6 · Missing data

A free, illustrated module of 4 lessons and one interactive in S01 Exploratory Data Analysis: Finding what is missing; Why values go missing: MCAR, MAR and MNAR; Dropping rows and columns; Filling in: simple imputation and its price.

S01.6 L1 · Core 4 lessons · about 100 min Live · in review

Part of S01 Exploratory Data Analysis in the Statistics and Data Analysis track.

Real tables have holes, and some of them hide behind numbers such as -999 or 0. These four lessons find every hole and count it, read the values that remain as a sample of the column that was wanted, and ask what kind of sample each reason for missingness gives: random, fair only inside groups, or bent by the hidden values themselves. They then price the two quick repairs, dropping incomplete rows and filling holes with a mean, so that you can say what each does to a mean, a spread and a correlation.

What you need first. The flats table and its columns (S01.1), population, sample, bias and the weighted mean (S01.2), and the mean, median and standard deviation (S01.3). Pearson’s correlation (S01.5) is used in the last two lessons. A little pandas helps for the code (P02).

Lessons

1. Finding what is missing S01.6.1 25 min

   A missing value is not always a NaN. It can be an empty cell, a -999 or a 0 that passes for data and moves every summary. Learn to unmask stand-ins, count the holes by column and by row, and read the pattern of holes as a picture.

2. Why values go missing: MCAR, MAR and MNAR S01.6.2 25 min

   The values you can see are a sample of the column you wanted. Learn the three reasons for a hole, missing completely at random, at random and not at random, what kind of sample each leaves, and why reweighting by a known column repairs one of them and not the other.

3. Dropping rows and columns S01.6.3 25 min

   Deleting incomplete rows is the default of most tools, and it costs more than it looks: ten columns with a tenth of their values missing keep only 35 percent of the rows. Derive that cost, see when deletion is a fair sample and when it is selection, and compare listwise, pairwise and column deletion.

4. Filling in: simple imputation and its price S01.6.4 25 min

   Filling every hole with the mean keeps all the rows and the mean, but it shrinks the spread by a factor you can compute, pulls correlations towards zero and plants a spike in the histogram. Derive each cost, soften them with group means and an indicator column, and fit the fill on training rows only.

The interactive

S01.6-X1 · Missingness machine. Thirty invented flats, area against rent. Choose how rents go missing and how many, then drop those rows or fill them with the mean, and compare the mean, the standard deviation and the correlation with those of the complete data. It lives in Why values go missing: MCAR, MAR and MNAR.