\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

S02.7 · Common tests: t, chi-square, ANOVA and nonparametric

A free, illustrated module of 4 lessons and one interactive in S02 Statistical Inference: t-tests: one sample, paired and two samples; Chi-square tests for counts; One-way ANOVA; Rank tests.

S02.7 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of S02 Statistical Inference in the Statistics and Data Analysis track.

Most questions a data scientist asks of a table are answered by a handful of named tests. These four lessons build each one from the same idea, a statistic compared with what chance alone would give: t-tests for one sample, pairs and two groups; chi-square tests for counts, with Fisher’s and McNemar’s exact tests; one-way ANOVA for several means; and rank tests for when normality is doubtful, ending with the AUC as a Mann-Whitney statistic.

What you need first. The t and chi-square distributions and the standard error (S02.1), confidence intervals (S02.4) and the logic of a test and the p-value (S02.5). Two-way tables of counts (S01.5.4) are used in the second lesson.

Lessons

1. t-tests: one sample, paired and two samples S02.7.1 25 min

   Every t-test divides a difference by its standard error and reads the result against a t curve. Run the one-sample, paired and two-sample tests by hand, see why pairing removes the spread between subjects, and know when to use Welch’s version.

2. Chi-square tests for counts S02.7.2 25 min

   Compare observed counts with the counts a hypothesis expects: the chi-square test of goodness of fit for a die, of independence for a two-way table, Fisher’s exact test for small tables, and McNemar’s test for two classifiers scored on one test set.

3. One-way ANOVA S02.7.3 25 min

   Compare several group means with one test: split the total sum of squares into a part between groups and a part within them, divide each by its degrees of freedom, and read their ratio F against the F distribution. Worked on rent by district and on the iris flowers.

4. Rank tests S02.7.4 25 min

   When normality is doubtful, replace values by their ranks. The sign test, the Wilcoxon signed-rank test, the Mann-Whitney U test and the Kruskal-Wallis test, with exact null distributions counted by hand, and why U divided by its maximum is a classifier’s AUC.

The interactive

S02.7-X1 · ANOVA decomposer. Rent for nine flats in three districts, each flat draggable or moved with a slider and buttons. A bar splits the total sum of squares into between and within, with F and its p-value; a switch replaces the rents by their ranks for the Kruskal-Wallis test, so dragging the penthouse shows one test react and the other hold. It lives in One-way ANOVA.