\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

S01.4 · Distributions and outliers

A free, illustrated module of 4 lessons and one interactive in S01 Exploratory Data Analysis: Histograms and the choice of bins; The cumulative view: ECDF and quantiles; Anatomy of a boxplot; Outliers: flag, check, decide.

S01.4 L1 · Core 4 lessons · about 100 min Live · in review

Part of S01 Exploratory Data Analysis in the Statistics and Data Analysis track.

A column of numbers has a shape, and these four lessons teach three honest ways to draw it and one careful way to deal with the values that sit far from the rest. The histogram counts values into bins whose width and starting point you choose; the staircase of the empirical distribution function needs no bins and reads off any share or quantile; the boxplot draws the five-number summary and marks the values beyond its fences. The last lesson compares three rules for flagging an unusual value, shows why the z-score cannot flag anything in a small sample, and sets out what to do with a flagged value: check it, keep it, report with and without it, or take logs.

What you need first. The mean, the median, quartiles and the five-number summary, the standard deviation and z-scores (S01.3), and counting a histogram in NumPy (P02.8.2). The last lesson uses base-10 logarithms (M01.4).

Lessons

1. Histograms and the choice of bins S01.4.1 25 min

   A histogram counts values into bins and draws a bar for each. Build one by hand for nine rents, see how the bin width and the starting point change the picture, and learn why densities, not counts, are the honest heights.

2. The cumulative view: ECDF and quantiles S01.4.2 25 min

   The empirical distribution function draws a column as a staircase that climbs by 1/n at every value. Build it by hand for nine rents, read the median, the quartiles and any share straight off it, and compare two groups with no bins to choose.

3. Anatomy of a boxplot S01.4.3 25 min

   A boxplot draws the five-number summary as a box, a median line and two whiskers, and marks every value beyond its fences one by one. Name every part, build one by hand for nine rents, and see what the box leaves out.

4. Outliers: flag, check, decide S01.4.4 25 min

   Three rules for flagging an unusual value disagree about the penthouse. Learn the fences, the z-score and the modified z-score, prove why no z-score among nine values can pass 2.67, and follow a habit for flagged values: check, keep, report with and without, or take logs.

The interactive

S01.4-X1 · Histogram bin-width explorer. Thirty-eight iris petal lengths as a rug of ticks with their histogram above. Slide the bin width and where the first bin starts, and watch two clumps appear, merge into one block or shatter into single flowers, with the number of bins and the Freedman-Diaconis width in the read-out. It lives in Histograms and the choice of bins.