A categorical column stores each label once and a small integer code per row. Build the look-up table and the codes for the sky column by hand, declare an order for the days, and turn codes into one-hot columns with broadcasting and pd.get_dummies.
P02.7.4L1 Core25 minIn review (what this means)
The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.
Build the categories and integer codes of a label column by hand, and match them with astype(‘category’), .cat.categories and .cat.codes.
Declare an ordered category so that sorting and comparisons follow the declared order, and explain why an unordered one refuses >.
Build a one-hot table by broadcasting codes[:, None] == np.arange(k), and with pd.get_dummies.
Explain, with distances, why bare codes invent an order between categories and one-hot columns do not.
The question
The sky column of the course’s readings holds twelve words: rain, rain, cloudy, clear, and so on. A model multiplies numbers by weights; it cannot multiply the word “cloudy”. The quick fix is to number the labels, clear 0, cloudy 1, rain 2. What does a table store when it does that, and why is that quick fix the wrong thing to hand to a model?
From the last lesson
1. How many distinct labels do ' Coast', 'HILL ', 'valley', 'coast ', 'hill', 'valley' have before and after .str.strip().str.lower()?
Show answer
Five before and three after: coast, hill and valley.
2. What does .str.extract(r"(\d+\.?\d*)", expand=False) give for 'n/a'?
Show answer
NaN: the pattern needs at least one digit, so there is no match.
Intuition
A column of words repeats itself. The sky column has twelve entries but only three different words. So store each word once, in a short look-up table, and store for each row only its position in that table: a small integer, its code. That is a categorical column.
P02.7.4-F1
A categorical column is a look-up table of labels plus one integer code per row. The codes point into the table.
The codes are a compact and exact copy of the column: look each code up in the table and the twelve words come back. But a code is a name, not an amount. Code 2 is not “twice” code 1, any more than room 2 of a hotel is twice room 1. Keeping that in mind is the whole of the last section of this lesson.
Build it up
Step 1. The table and the codes by hand. The categories are the distinct labels, sorted: sorted(set(sky)) is ['clear', 'cloudy', 'rain']. A dict gives each its position, {'clear': 0, 'cloudy': 1, 'rain': 2}, and a comprehension looks every row up in it.
Step 2. The same in pandas.sky.astype("category") makes the categorical column. Its .cat.categories is the table and its .cat.codes is the codes, stored as int8, one byte each. A missing value gets the code −1.
Step 3. Back again. Indexing the table with the codes, categories[codes], rebuilds the column. Nothing was lost.
Step 4. One column per category. For a model, turn the one column of codes into three columns of 0 and 1, one per category, with a 1 in the column of the row’s label. This is one-hot encoding. pd.get_dummies does it for a whole table, and leaves the number columns as they are.
P02.7.4-F5
pd.get_dummies replaces the text column by one 0/1 column per category. The table a model receives is all numbers.
A categorical column with \(k\) categories stores a table of \(k\) distinct labels, numbered \(0\) to \(k - 1\), and for each row a code, the number of its label. pandas sorts inferred categories, and uses the code −1 for a missing value.
A categorical is ordered if its categories carry a declared order. Then sorting follows that order, and <, >, min and max are allowed. For an unordered categorical they raise a TypeError.
The one-hot encoding of a column with codes \(c_0, \dots, c_{n-1}\) is the table \(E\) with \(n\) rows and \(k\) columns whose entry in row \(i\), column \(j\) is 1 if \({c_i = j}\) and 0 if not.
Fact 1: each row of \(E\) has exactly one 1. In a column with no missing values, a row’s code \(c_i\) is one of \(0, \dots, k - 1\), so it equals exactly one column number \(j\). (A missing value, code −1, matches no column, and its row is all 0.) Summing a column \(j\) therefore counts the rows with code \(j\): the column sums of \(E\) are the class counts.
Fact 2: one-hot is a broadcast. Put the codes in a column of shape \((n, 1)\) and the numbers \(0, \dots, k - 1\) in a row of shape \((1, k)\). Comparing them with == stretches both to \((n, k)\), by the rule of P02.2.3, and compares entry by entry, so the result in row \(i\), column \(j\) is \({c_i = j}\): exactly \(E\), written True and False.
Fact 3: one-hot rows are all the same distance apart. The row of category \(a\) is the vector \(\vect{e}_a\) with a 1 in place \(a\) and 0 elsewhere. For two different categories \(a\) and \(b\), the difference \(\vect{e}_a - \vect{e}_b\) has a 1, a −1 and zeros, so
and every pair is \(\sqrt{2}\) apart. With bare codes the distances are \(\abs{a - b}\): clear is 1 from cloudy but 2 from rain, an order and a spacing that nothing in the weather supplies.
P02.7.4-F2
Declaring an order. Inferred categories are sorted alphabetically and have no order; declared ones sort and compare in the order given.
The day labels show why the order matters. Sorted as text, they come out mon, thu, tue, wed, the order that P02.5.3’s pivot produced. Declared as an ordered category with the order mon, tue, wed, thu, they sort by the week, and day > "tue" is True for Wednesday and Thursday.
Worked example
Encode the sky column of the 12 readings, by hand.
Codes. The categories, sorted, are clear, cloudy, rain, with codes 0, 1, 2. The sky column reads rain, rain, cloudy on Monday; clear, clear, cloudy on Tuesday; clear, cloudy, clear on Wednesday; and clear, cloudy, clear on Thursday. Looking each up gives the codes 2, 2, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0.
P02.7.4-F3
One-hot by broadcasting, shown for the first six rows. Each row’s code meets the row 0, 1, 2 and is True in exactly one place.
One-hot. Row 0 has code 2, so it compares to the row 0, 1, 2 as False, False, True. Row 2 has code 1: False, True, False. Every row has exactly one True (Fact 1). Down the columns, clear has a True in rows 3, 4, 6, 8, 9 and 11, six in all; cloudy in rows 2, 5, 7 and 10, four; rain in rows 0 and 1, two. The column sums 6, 4, 2 are the counts of P02.6.2, here as a by-product of the encoding.
Back again. The table indexed by the codes, ['clear', 'cloudy', 'rain'][2] for row 0 and so on, returns rain, rain, cloudy, …: the column we began with.
Code
By hand, then with pandas.
import numpy as npimport pandas as pdsky = pd.Series(["rain", "rain", "cloudy", "clear", "clear", "cloudy","clear", "cloudy", "clear", "clear", "cloudy", "clear"])categories =sorted(set(sky))lookup = {name: k for k, name inenumerate(categories)}codes = [lookup[s] for s in sky]print(categories, codes)cat = sky.astype("category")print(cat.cat.categories.tolist(), cat.cat.codes.tolist(), cat.cat.codes.dtype)print(cat.cat.categories[cat.cat.codes].tolist() == sky.tolist())
This module’s interactive, Resample and roll, lives in lesson P02.7.2. For this lesson, change the order of the categories in week and rerun the last cell: sort_values and day > "tue" follow whatever order you declare.
Pitfalls
P02.7.4-F4
Wrong beside right. Bare codes put the sky on a line, with cloudy between clear and rain. One-hot columns put the three labels at the corners of a triangle with equal sides.
“The codes are numbers, so the model can use them directly.”
A linear model would give the sky column one weight \(w\) and add \({w \times \text{code}}\) to its prediction. Then rain adds twice what cloudy adds, and cloudy must lie between clear and rain, whatever the data say. scikit-learn’s user guide warns of exactly this: integer codes would be interpreted as ordered categories, which is often not desired. One-hot columns give each category its own weight. Bare codes are fine only for an order that is real, such as the days of the week or sizes S, M, L, and even then the gaps are assumed equal.
“The test set will get the same columns.”
pd.get_dummies makes one column per category present. A test set with no rain gets two columns, not three, and a model trained on three cannot use it. Fix the categories first, test_sky.astype(pd.CategoricalDtype(["clear", "cloudy", "rain"])), and get_dummies then makes all three columns, the rain column all 0. The categories, like any statistic, come from the training data.
“A category column takes any new value.”
Setting an entry to a label that is not among the categories, such as "snow", raises a TypeError. Add the category first with .cat.add_categories, or keep the column as text until the set of labels is settled.
Check yourself
Recall
1. What two things does a categorical column store?
Show answer
A table of the distinct labels, the categories, each stored once, and for each row an integer code, the position of its label in the table.
2. What are the categories and codes of pd.Series(["b", "a", None]).astype("category")?
Show answer
Categories ['a', 'b'], sorted; codes 1, 0 and −1, where −1 marks the missing value.
3. Why does cat > "clear" raise an error on the sky column, but day > "tue" work on the days?
Show answer
The sky categories are unordered, and unordered categories refuse >. The days were declared an ordered category, so comparisons follow the declared order.
4. What does codes[:, None] == np.arange(3) compute, and what shape is it for 12 rows?
Show answer
The one-hot table, True where a row’s code equals the column number. The column (12, 1) and the row (1, 3) broadcast to (12, 3).
5. How far apart are two different one-hot rows, and why does that matter?
Show answer
\(\sqrt{2}\), for every pair, so no category is nearer to one than to another. Bare codes are 1 or 2 apart, which invents an order and a spacing.
Exercises
1.★ Encode the station column of the readings, coast, hill, valley repeated four times, by hand: give the categories, the first six codes and the class counts.
Show solution
The categories, sorted, are coast, hill, valley, with codes 0, 1, 2. The column repeats coast, hill, valley, so the codes are 0, 1, 2, 0, 1, 2, and so on. Each station appears 4 times, so every column of the one-hot table sums to 4.
2.★★ A linear model uses the bare sky code with weight \(w = 1.5\). How much does it add to the prediction for clear, cloudy and rain? Then give the one-hot version with weights \(w_{\text{clear}} = 0.5\), \(w_{\text{cloudy}} = -1\), \(w_{\text{rain}} = 3\), and say which pattern the bare code could never express.
Show solution
With the bare code it adds 1.5 × 0 = 0 for clear, 1.5 × 1 = 1.5 for cloudy and 1.5 × 2 = 3 for rain: always in steps of equal size, so cloudy is forced to sit exactly halfway between clear and rain. With one-hot columns the row for cloudy is (0, 1, 0), so the model adds \({0.5 \times 0} + {(-1) \times 1} + {3 \times 0} = -1\); clear adds 0.5 and rain adds 3. Cloudy now adds less than both, a pattern no single weight on the code can give, since \({1.5 \times 1}\) always lies between \({1.5 \times 0}\) and \({1.5 \times 2}\).
3.★★★ The readings of Monday and Tuesday (rows 0 to 5) are the training set and Wednesday’s (rows 6 to 8) are the test set. Show that pd.get_dummies gives the two sets different columns, and fix it so that both get the same columns, with the categories taken from the training set only.
Show solution
The training sky is rain, rain, cloudy, clear, clear, cloudy, which holds all three labels, so get_dummies gives it the columns clear, cloudy and rain. Wednesday’s sky is clear, cloudy, clear, with no rain, so it gets only clear and cloudy: two columns where the model expects three. Fix the categories from the training set before encoding: sky_type = pd.CategoricalDtype(sorted(train.unique())), then pd.get_dummies(train.astype(sky_type), dtype=int) and the same for test. Both tables now have the columns clear, cloudy, rain in that order, and the test table’s rain column is all 0. A label that appears only in the test set is not among the categories and becomes missing, which is worth checking for before training.
Summary card
Store labels once with integer codes; give a model one-hot columns, not the codes.
P02.7.4-F1
A look-up table plus one code per row.
s.astype("category"): .cat.categories (sorted labels) and .cat.codes (int8, −1 if missing).
pd.CategoricalDtype(order, ordered=True): sorting and <, > follow the order.
One-hot: codes[:, None] == np.arange(k), or pd.get_dummies; column sums are class counts.
One-hot rows are all \(\sqrt{2}\) apart; bare codes invent an order.
Fix the categories from the training data, so train and test get the same columns.
Sources and next steps
Sources. Each was opened and checked against this lesson on 15 November 2026.
pandas developers, Categorical data, pandas 3.1.1 User Guide. A categorical consists of a categories array and an integer array of codes; missing values always have the code −1; ordered categoricals sort by the order of the categories, and an unordered one raises a TypeError for min and max; setting a value that is not among the categories raises a TypeError.
pandas developers, pandas.get_dummies, pandas 3.1.1 API reference. Converts a categorical variable into dummy or indicator variables, one per distinct value, with bool as the default dtype, and drop_first to keep \(k - 1\) of \(k\) levels.
scikit-learn developers, Encoding categorical features, scikit-learn 1.9.1 User Guide, section 8.3.4. Integer codes from OrdinalEncoder would be interpreted by estimators as ordered categories, which is often not desired; OneHotEncoder gives one-of-K (one-hot or dummy) encoding, with one binary feature per category.
Next. P02.8 Plotting with Matplotlib and seaborn draws these tables: the categories become the colours of a plot and the dates its time axis.
![Above, the code codes[:, None] == np.arange(3). Left, the first six codes 2, 2, 1, 0, 0, 1 as a blue column of shape (12, 1), with two dashed ghost copies beside it, stretched to (12, 3). Then an equals-equals sign, and the row 0, 1, 2 in sky blue, shape (1, 3), with dashed ghost copies down the rows, stretched to (12, 3). A green arrow leads to the vermillion result of shape (12, 3), with T for True shaded yellow and F for False: one T per row, in the rain column for the two rain rows, the cloudy column for the cloudy row, and so on, each row labelled with its sky word. Underneath: over all 12 rows the column sums are 6, 4, 2, the class counts of clear, cloudy and rain.](figures/P02.7.4-F3.svg)
