\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML02.11 · Choosing a model for tabular data

A free, illustrated module of 4 lessons and one interactive in ML02 Supervised Learning: Trees, Ensembles and Kernels: Every model makes an assumption; Matching the model to the data; A fair comparison; Case study: the diabetes project revisited.

ML02.11 L2 · Intermediate 4 lessons · about 100 min Live · in review

Part of ML02 Supervised Learning: Trees, Ensembles and Kernels in the Machine Learning track.

The course has built trees, forests, boosting, naive Bayes, discriminant analysis and kernel machines. Which one should you use on a new table? These four lessons answer in three parts. Every model carries an assumption about the shape of the answer, and wins where the data have that shape. A checklist read off the table itself, its rows, columns, types, gaps and scales, narrows the field before anything is fitted. Then a fair comparison on the same repeated folds, with honest error bars, decides between what is left. The module ends by rerunning the diabetes project of ML01.10 with every model of the course.

What you need first. Decision trees and their ensembles (ML02.1 to ML02.5), naive Bayes (ML02.6), support vector machines and kernels (ML02.7 and ML02.8), k-nearest neighbours and logistic regression (ML01.3 and ML01.6), and cross-validation with standard errors and the one-standard-error rule (ML01.9 and ML01.10). The variance of an average of correlated quantities (ML02.2.1) is used in the third lesson.

Lessons

1. Every model makes an assumption ML02.11.1 25 min

   A tree assumes boxes, a linear model one straight cut, k-nearest neighbours that near points agree, naive Bayes one feature at a time. Fit all four to the same four datasets and see each assumption help on one and hurt on another.

2. Matching the model to the data ML02.11.2 25 min

   Before fitting anything, read the table: its size, its column types, its gaps, its units, and what the answer must be used for. A seven-question checklist, each family scored against it, and what the best-known benchmark of tree ensembles does and does not show.

3. A fair comparison ML02.11.3 25 min

   Score every candidate on the same repeated stratified folds, compare them fold by fold, and correct the standard error for the overlap between folds. A difference smaller than that error decides nothing, and the best of many tuned models is always flattered.

4. Case study: the diabetes project revisited ML02.11.4 25 min

   The diabetes project of ML01.10 rerun with every model of this course: a pruned tree, a random forest, gradient boosting, SVR and kernel ridge against the linear models, on the same patients and the same repeated folds, ending in a choice and a one-page report.

The interactive

ML02.11-X1 · Model line-up. Pick a dataset, a diagonal line, the moons, the rings or two blobs with a noise feature, and see a tree, k-nearest neighbours, logistic regression and naive Bayes draw their boundaries side by side, each with its accuracy on fresh points. Slide the training-set size and watch which assumptions pay off with little data. It lives in Every model makes an assumption.