\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML01.4 · Linear regression as a learning algorithm

A free, illustrated module of 4 lessons and one interactive in ML01 Machine Learning Foundations: A line as a model; The best line, solved exactly; Many features; Reading and doubting a fitted line.

ML01.4 L1 · Core 4 lessons · about 115 min Live · in review

Part of ML01 Machine Learning Foundations in the Machine Learning track.

A straight line is the simplest model that predicts a number, and fitting one shows every part of machine learning in miniature: a model with parameters, a loss that scores it, and a way to find the best parameters. These four lessons score a line by its squared misses, solve for the best line exactly, extend the solution to any number of features with one matrix equation, and then teach you to read a fitted line and to doubt it.

What you need first. A model as a function with parameters and a loss that scores it (ML01.1), and straight lines and their graphs (M01.3). The second lesson uses the chain rule and the search for a lowest point (M04.3, M04.4) and a first look at partial derivatives (M04.10); the third uses matrix multiplication and linear systems (M02.5, M02.6).

Lessons

1. A line as a model ML01.4.1 25 min

   A straight line with a slope and an intercept is the simplest model that predicts a number. Learn what a residual is, why the misses are squared before they are averaged, and how the mean squared error and the mean absolute error judge the same four points differently.

2. The best line, solved exactly ML01.4.2 30 min

   Set both partial derivatives of the mean squared error to zero and solve, one step at a time. The best slope is the covariance of x and y divided by the variance of x, and the best line passes through the point of means. Worked by hand on the course’s four points.

3. Many features ML01.4.3 30 min

   With several features, every prediction is a row of a table times a column of weights: X times w. Derive the normal equations, solve them by hand for six flats and with NumPy’s lstsq, see how a column of ones carries the bias, and learn why X transpose X should not be inverted.

4. Reading and doubting a fitted line ML01.4.4 30 min

   A fitted line hands you a few numbers: a slope, an intercept, an R squared. Learn what a coefficient means and which units it carries, derive R squared as the share of variance the line explains, and use residual plots, outliers and Anscombe’s quartet to see when the numbers mislead.

The interactive

ML01.4-X1 · Line fitter. Drag the line or the points and watch the residuals and their squares change. The read-out gives the mean squared error and the mean absolute error of your line, and the best-fit line it is competing against. It lives in A line as a model.