\( \newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML01.6 · Logistic regression and classification

A free, illustrated module of 4 lessons and one interactive in ML01 Machine Learning Foundations: From a line to a probability; The log loss; Training by gradient descent; Decision boundaries and thresholds.

ML01.6 L1 · Core 4 lessons · about 100 min Live · in review

Part of ML01 Machine Learning Foundations in the Machine Learning track.

Many questions have a yes-or-no answer: will this student pass, is this email spam, is this tumour malignant? A straight line is the wrong tool for them, and logistic regression is the right first one. These four lessons bend the line into an S-shaped curve that outputs a probability, derive the log loss that scores such a curve from the likelihood of the data, train it by gradient descent with a gradient that turns out to be prediction minus label, and then turn its probabilities into decisions with a boundary and a threshold.

What you need first. A line as a model and the mean squared error (ML01.4), gradient descent and the learning rate (ML01.5), exponentials and logarithms (M01.4), and the chain rule (M04.3). The idea that independent probabilities multiply (M06.3) is used once, in lesson 2.

Lessons

From a line to a probability ML01.6.1 25 min

A straight line is a poor model for a yes-or-no label: it predicts numbers below 0 and above 1. The sigmoid squashes any score into a probability, and the model that results is still a straight line, in the log-odds.
The log loss ML01.6.2 25 min

Score a probability curve by how probable it makes the labels that were actually seen. Taking logarithms turns that product into a sum, the mean negative log-likelihood or cross-entropy, and shows why squared error is the wrong loss for a sigmoid.
Training by gradient descent ML01.6.3 25 min

Differentiate the sigmoid, then the log loss, and the gradient collapses to prediction minus label, times the feature. Train logistic regression from scratch, match scikit-learn’s LogisticRegression, and see what its default regularisation changes.
Decision boundaries and thresholds ML01.6.4 25 min

A probability becomes a decision when it is compared with a threshold. The border between the two answers is where the score is zero: a straight line in two features. Moving the threshold trades false alarms for misses, and softmax extends the model to many classes.

The interactive

ML01.6-X1 · Sigmoid and boundary explorer. Set the weight and the bias of a logistic regression by hand and watch the probability curve, the log loss of every student and the mistakes at your chosen threshold. Then press Train and let gradient descent find the best curve, for one feature or for two. It lives in Training by gradient descent.