\( \newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\mat}[1]{\boldsymbol{\mathbf{#1}}} \newcommand{\R}{\mathbb{R}} \newcommand{\E}{\mathbb{E}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\T}{^{\top}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\ihat}{\mkern1mu\hat{\boldsymbol{\imath}}\mkern2mu} \newcommand{\jhat}{\mkern1mu\hat{\boldsymbol{\jmath}}\mkern2mu} \newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\cdata}[1]{\class{c-data}{#1}} \newcommand{\cmodel}[1]{\class{c-model}{#1}} \newcommand{\ctruth}[1]{\class{c-truth}{#1}} \newcommand{\cgrad}[1]{\class{c-grad}{#1}} \newcommand{\closs}[1]{\class{c-loss}{#1}} \newcommand{\cparam}[1]{\class{c-param}{#1}} \newcommand{\csecond}[1]{\class{c-second}{#1}} \renewcommand{\,}{\mkern3mu} \)

ML02.5 · XGBoost, LightGBM and CatBoost

A free, illustrated module of 4 lessons and one interactive in ML02 Supervised Learning: Trees, Ensembles and Kernels: Second-order boosting: the XGBoost objective; Histograms and leaf-wise growth: LightGBM’s ideas; Categorical features without leakage: CatBoost’s ideas; The three libraries side by side.

ML02.5 L2 · Intermediate 4 lessons · about 120 min Live · in review

Part of ML02 Supervised Learning: Trees, Ensembles and Kernels in the Machine Learning track.

The three libraries that win most contests on tables of data are gradient boosting with three good ideas added. XGBoost takes a second-order step and puts a price on every leaf, so that one formula, G squared over H plus lambda, scores both a leaf and a split. LightGBM bins each feature once and grows the most promising leaf first. CatBoost turns a categorical column into numbers without letting a row see its own label. These four lessons derive each idea, build it from scratch in NumPy, check it against scikit-learn’s HistGradientBoosting estimators, and end with one map of the settings that the four libraries share under different names.

What you need first. Gradient boosting as fitting residuals and as gradient descent in function space, with the Newton leaf value of boosting for classification (ML02.4), regression trees grown greedily (ML02.1.3), and a penalty on the size of weights (ML01.9). Cross-validation (ML01.9.3) is used in the last two lessons.

Lessons

1. Second-order boosting: the XGBoost objective ML02.5.1 30 min

   XGBoost adds a price on leaves to the boosting loss and replaces each loss by its parabola. Then every leaf is a parabola in its weight, and one formula, G squared over H plus lambda, gives the leaf weight, the score of a tree and the gain of a split.

2. Histograms and leaf-wise growth: LightGBM’s ideas ML02.5.2 30 min

   Bin each feature once, and the search for the best split becomes a pass over a few hundred bin sums of g and h, with one child’s histogram got free by subtraction. Then grow the most promising leaf first, and let missing values learn which way to go.

3. Categorical features without leakage: CatBoost’s ideas ML02.5.3 30 min

   Replacing a category by the mean label of its rows looks like a gift and is a trap: each row’s own label leaks into its feature. See the leak on a seeded table, then fix it with CatBoost’s ordered target statistics, cross-fitting, and native categorical splits.

4. The three libraries side by side ML02.5.4 30 min

   XGBoost, LightGBM, CatBoost and scikit-learn’s HistGradientBoosting share one objective under different names. Map their settings onto one another, see where each enters the formulas, compare them fairly on two datasets, and learn which settings to change first.

The interactive

ML02.5-X1 · Split-gain calculator. Twelve points along one feature, with their gradients and Hessians drawn as bars. Slide the threshold, lambda and gamma and watch the sums G and H on each side, the two leaf weights and the gain change; then cut the number of bins and see the thresholds shrink to histogram edges. It lives in Histograms and leaf-wise growth: LightGBM’s ideas.