\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

M06.9 · Joint, marginal and conditional distributions

A free, illustrated module of 4 lessons and one interactive in M06 Probability I: Joint distributions as tables; Marginals: summing out a variable; Conditional distributions and independence; Joint densities.

M06.9 L1 · Core 4 lessons · about 100 min Live · in review

Part of M06 Probability I in the Mathematics track.

Two random variables read off the same experiment are tied together, and their separate PMFs cannot show how. These four lessons put the pair on one table, the joint distribution, and read three things from it: each variable’s own distribution by summing out the other, the distribution of one given a value of the other by rescaling a row, and independence as every cell being the product of its margins. The last lesson does the same with a density over the plane, where probability is a volume and sums become an integral inside an integral.

What you need first. Discrete random variables, their PMFs and independent pairs (M06.5), conditional probability and the law of total probability (M06.3), and densities and integrals (M06.6, M04.5 to M04.7) for the last lesson. The code uses fractions, itertools and Counter, NumPy arrays with axis sums and broadcasting, pandas.crosstab and scipy.integrate.dblquad.

Lessons

1. Joint distributions as tables M06.9.1 25 min

   Two random variables on the same outcomes have a joint PMF: one probability for every pair of values, laid out as a table. Build the course’s table from three coin flips, read any event about the pair as a sum of cells, and count the table in NumPy and pandas.

2. Marginals: summing out a variable M06.9.2 25 min

   The PMF of one variable is hidden in the joint table: add each row for X and each column for Y. Why that is the law of total probability, why the margins cannot rebuild the table, and how summing out a hidden variable gives the mixture models of machine learning.

3. Conditional distributions and independence M06.9.3 25 min

   Learning the value of X picks out one row of the joint table; rescaled to add to 1, that row is the conditional PMF of Y, with its own mean. The product rule, Bayes’ rule for PMFs, and independence as every row rescaling to the same PMF.

4. Joint densities M06.9.4 25 min

   For two continuous variables the joint table becomes a surface with volume 1 beneath it, and the probability of a region is the volume above it: an integral inside an integral. Marginals, conditionals and independence carry over from the table, worked on a square and a triangle.

The interactive

M06.9-X1 · Joint table explorer. The course’s 3 by 3 joint table as shaded cells, with the two marginals as bars along its edges. Pick a row or a column to condition on and watch that slice rescale into a conditional PMF with its mean marked; a toggle swaps in the independent table with the same marginals, where every slice rescales to the same PMF. It lives in Conditional distributions and independence.