\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

M06.10 · Covariance and correlation

A free, illustrated module of 4 lessons and one interactive in M06 Probability I: Covariance: do they move together?; Rules of covariance and the variance of a sum; Correlation: covariance without units; What correlation does not tell you.

M06.10 L1 · Core 4 lessons · about 100 min Live · in review

Part of M06 Probability I in the Mathematics track.

Two random variables can rise and fall together, pull in opposite directions, or ignore each other. These four lessons measure that with two numbers. The covariance is the average product of the two deviations from the means, pictured as rectangles that are positive in two quadrants and negative in the other two. Its rules give the variance of any sum, which is why an ensemble wants members whose errors do not move together. The correlation strips out the units, always lies between −1 and 1, and is the slope of the best line in standard units. The last lesson is about what correlation cannot see: curves, outliers and shared causes.

What you need first. Expectation, variance and standardising (M06.7), joint tables with their marginals and conditional means (M06.9), and Python with fractions and NumPy. The code ends with numpy.cov, numpy.corrcoef and scipy.stats.pearsonr.

Lessons

1. Covariance: do they move together? M06.10.1 25 min

   Covariance is the average product of two deviations from the means: positive in two quadrants, negative in the other two. Derive the shortcut E[XY] − E[X]E[Y], work it on the course’s joint table, and see why its size depends on the units.

2. Rules of covariance and the variance of a sum M06.10.2 25 min

   Covariance is symmetric and linear in each slot. Derive those rules, then the variance of any sum, Var(X) + Var(Y) + 2 Cov(X, Y), find the course’s covariance again from the flip the two counts share, and see why an ensemble wants errors that do not move together.

3. Correlation: covariance without units M06.10.3 25 min

   Correlation is the covariance of the two standardised variables, so no change of units can move it. One short calculation shows that it lies between −1 and 1, equals ±1 only for a straight line, and is the slope of the best line in standard units.

4. What correlation does not tell you M06.10.4 25 min

   Independent variables have covariance 0, but zero covariance does not mean independence. Correlation sees only straight-line trends, one point can flip its sign, a shared cause creates it without causation, and from a few data points it strays a long way.

The interactive

M06.10-X1 · Correlation explorer. Thirty people as a cloud of points, cut into four quadrants at the means, with each point’s rectangle of deviations shaded by its sign. Set the correlation with a slider, write the heights in centimetres instead of metres, and try a curved cloud and a single outlier: the read-out gives the covariance and the correlation and says which one moved. It lives in Correlation: covariance without units.