7 Encoding and Features
Models take numbers. Categories, words and the bends in a numeric relationship have to become numbers first, and the choice of encoding decides which functions of them are easy to learn. An encoding can assert an order the data does not have, merge categories that should stay apart, or, in one popular case, copy the label into the features. This chapter treats the common encodings as choices on the representation lever, and measures the one that leaks.
7.1 The problem
Given a categorical column, a numeric column whose effect is curved, or a column of text, which representation lets the model learn the right function, and what does each representation assume? For one encoding, target encoding, there is a second question. It is computed from the labels, so how should it be fit without leaking them?
7.2 Assumptions
- The model is fixed; only the representation changes. Each demonstration holds the model constant (an unpenalized logistic regression or a linear regression) and changes only the encoding, so differences come from the representation.
- Levels and vocabularies are learned from training data only. A level or a word first seen at test time has no column, unless the encoder reserves one or hashes it.
- For target encoding, rows are exchangeable. With rows ordered in time, the encoding must also respect time (Data Types and Their Structure).
7.3 Derivation or demonstration
One-hot and ordinal codes. A one-hot encoding gives each level its own indicator column; an ordinal encoding gives each level an integer. A linear model on one-hot columns can assign every level its own effect. On an ordinal code it can fit only a straight line in the code, asserting both an order and an equal spacing. In the demonstration, a category has six levels whose effects (0, 3, −2, 1, −3, 2) are unrelated to their codes. Linear regression on the one-hot encoding reached a test \(R^2\) of 0.947, equal to the ceiling the noise allows. On the ordinal code it reached 0.008 (tests/test_encoding.py::test_ordinal_codes_impose_an_order_one_hot_does_not).
| Changes | Representation (of a categorical \(x\)): one indicator column per level (one-hot), or one integer code (ordinal) |
| Assumption | One-hot: none about order; ordinal: the levels are ordered and the code’s spacing means something to the model |
| What it does to the learned function | One-hot lets a linear model give every level its own effect; an ordinal code forces a linear model to treat the effect as a straight line in the code |
| Which models care | Linear and distance-based models depend on the choice; a tree can separate any level of an ordinal code, but one interval of codes per split, so unordered levels cost extra splits |
| Fit on | Training data only: the list of levels |
| Failure modes | Ordinal codes for unordered categories; unseen levels at test time; one-hot with very many levels (wide, sparse, rare columns) |
| Alternatives | Target encoding (cross-fitted); hashing; grouping rare levels |
| Checked by | tests/test_encoding.py::test_ordinal_codes_impose_an_order_one_hot_does_not |
Hashing. With very many levels, or an open-ended set (identifiers, words), feature hashing maps each level to one of \(m\) columns by a hash function, with no stored vocabulary (Weinberger et al. 2009). Distinct levels that land in the same column share it. For a uniform hash, \(k\) levels occupy on average \(m\,(1 - (1 - 1/m)^k)\) of the \(m\) columns. Hashing 2,000 levels with scikit-learn’s FeatureHasher used 880 of 1,024 columns (expected 879) and 1,775 of 8,192 (expected 1,775). Even with four times as many columns as levels, the 2,000 levels fell into only 1,775 columns, so at least 226 of them share a column with another (tests/test_encoding.py::test_feature_hashing_collides_at_the_expected_rate).
| Changes | Representation: each level (or token) is mapped by a hash function to one of \(m\) columns, with no stored vocabulary |
| Assumption | Collisions between levels are rare enough, or harmless enough, to accept for fixed memory |
| What it does to the learned function | Like one-hot, except that colliding levels share a column and so share an effect |
| Which models care | Linear models on very many or open-ended categories (text, identifiers) |
| Fit on | Nothing: the hash is fixed |
| Failure modes | Collisions grow with the number of levels: 2,000 levels in 1,024 columns use only about 880 of them; hashed columns cannot be traced back to levels |
| Alternatives | One-hot with a vocabulary; grouping rare levels; target encoding |
| Checked by | tests/test_encoding.py::test_feature_hashing_collides_at_the_expected_rate |
Target encoding. A high-cardinality category can instead be replaced by one number per level: the mean target among training rows with that level, shrunk toward the global mean for rare levels. scikit-learn’s TargetEncoder follows Micci-Barreca’s scheme of mixing the global mean with the level’s mean (Micci-Barreca 2001). The encoding is computed from the labels, which makes it a supervised step, and Cleaning showed that supervised steps fit on the evaluation rows fabricate signal. Here the evaluation rows are the training rows themselves. If each row’s encoding includes its own label, the feature carries that label into the model.
The demonstration adds one pure-noise feature to T1’s classification task: a category with 1,000 levels, assigned at random. Over 20 seeds, with 2,000 training rows:
- without the feature, training and test accuracy were 0.746 and 0.752;
- encoded on the same rows it was fit on, training accuracy rose to 0.854 and test accuracy fell to 0.618. The model put a large coefficient (7.8, on a probability-scale feature) on noise;
- cross-fitted, where each row is encoded by an encoder fit on the other folds, as
TargetEncoder.fit_transformdoes (here with a seeded 5-fold splitter), the result was 0.747 and 0.751, with a coefficient of \(-0.05\) (tests/test_leakage.py::test_target_encoding_without_cross_fitting_leaks).
A target encoding computed on the same rows it encodes includes each row’s own label. With many rare levels that is enough to turn pure noise into the strongest feature in training, and the model is then worse on new data than without it. Cross-fitting, encoding each row with statistics from the other folds, removes the leak.
| Changes | Representation: each level is replaced by a shrunk estimate of the mean target among training rows with that level |
| Assumption | Levels with few rows are shrunk toward the global mean; each row’s encoding is computed without that row’s own label (cross-fitting) |
| What it does to the learned function | One informative column per categorical feature, however many levels; without cross-fitting, each row’s own label leaks into its feature |
| Which models care | Every model; most useful for high-cardinality categories with linear or tree models |
| Fit on | Training labels, cross-fitted on the training rows (out-of-fold), and fit on all training rows for test data |
| Failure modes | Encoding training rows with an encoder fit on those same rows: noise becomes a training signal that fails on new data; leakage across a time order |
| Alternatives | One-hot; hashing; grouping rare levels |
| Checked by | tests/test_leakage.py::test_target_encoding_without_cross_fitting_leaks |
Splines and bins. A numeric feature whose effect is curved defeats a linear model on the raw value. Two encodings give the linear model a curve to fit. Discretization replaces \(x\) by indicators of intervals, and splines replace it by a basis of piecewise polynomials. On \(y = \sin(2x)\) plus noise, a linear regression on raw \(x\) reached a test \(R^2\) of 0.14. On 20 quantile bins it reached 0.82, and on a cubic spline basis with 8 knots 0.85, against a noise ceiling of 0.85 (tests/test_encoding.py::test_splines_and_bins_let_a_linear_model_fit_a_curve).
| Changes | Representation (of a numeric \(x\)): a basis of piecewise polynomials (splines) or of interval indicators (bins) |
| Assumption | The effect of \(x\) is smooth (splines) or roughly constant within intervals (bins); enough data per knot or bin |
| What it does to the learned function | A linear model can fit a curve in \(x\); bins give a step function, splines a smooth one |
| Which models care | Linear and GLM-type models; trees already cut a feature into intervals |
| Fit on | Training data only: knot or bin positions (quantiles or a uniform grid) |
| Failure modes | Too many knots or bins (variance); extrapolation beyond the training range; bins hide variation within an interval |
| Alternatives | A shape-changing transform; a model that is nonlinear in \(x\) itself |
| Checked by | tests/test_encoding.py::test_splines_and_bins_let_a_linear_model_fit_a_curve |
Text: from counts to TF-IDF. The simplest representation of a document counts its words, discarding order. TF-IDF reweights the counts so that words common to every document count less. In scikit-learn’s default it multiplies the count \(\text{tf}(t, d)\) of term \(t\) in document \(d\) by \(\ln\big((1 + n)/(1 + \text{df}(t))\big) + 1\), where \(n\) is the number of documents and \(\text{df}(t)\) the number containing \(t\), then scales each document’s vector to unit length (scikit-learn 1.9, TfidfTransformer docstring; a test reproduces its output from this formula to 12 digits: tests/test_encoding.py::test_tfidf_matches_the_documented_formula). Learned embeddings replace this with vectors trained on other text, which belongs to the architecture books.
| Changes | Representation (of a document): term counts, weighted down for terms that occur in many documents, scaled to unit length |
| Assumption | Word order does not matter for the task; terms common to every document carry little information |
| What it does to the learned function | A fixed-length vector in which rare, distinctive terms dominate and document length matters less |
| Which models care | Linear models and distance-based (cosine) models on text |
| Fit on | Training documents only: the vocabulary and document frequencies |
| Failure modes | Order and context are lost; rare terms are noisy; the vocabulary is fixed at fit time; the default token pattern drops one-character tokens |
| Alternatives | Raw counts; hashing; learned embeddings (beyond this book) |
| Checked by | tests/test_encoding.py::test_tfidf_matches_the_documented_formula |
Images and audio. Pixel intensities and audio samples are already numbers. Their encoding questions are the representation and scale questions of the previous chapters: rescaling intensities is affine (Affine Scaling), and a spectrogram is a change of representation (Data Types and Their Structure).
In scikit-learn 1.9, TargetEncoder().fit_transform(X, y) cross-fits (cv=5), but fit(X, y).transform(X) does not; its documentation notes that the second “does not equal” the first. Use fit_transform on training data and transform on new data, or put the encoder in a Pipeline. Its folds are shuffled without a fixed seed by default, so pass a splitter such as cv=KFold(5, shuffle=True, random_state=0) for reproducible results (the shuffle and random_state arguments are deprecated in 1.9). TfidfVectorizer’s default token_pattern is (?u)\b\w\w+\b: tokens of two or more word characters (Unicode letters, digits or the underscore), so one-character words are dropped.
7.4 What it does to the learned model
An encoding sets the hypothesis space a fixed model can reach. One-hot columns let a linear model fit any per-level effect; an ordinal code limits it to a line in an arbitrary order. Splines let it fit smooth curves. Hashing lets colliding levels share an effect. Target encoding compresses a category into one number, which is efficient when the number is honest and destructive when it contains the label. In the last case the model learns to read the leak, and on new data, where there is none, it is worse than if the feature had been left out.
7.5 Failure modes
Figure 7.1 shows the two central demonstrations. Further failures:
- Unseen levels. A level absent from training has no one-hot column and no target mean. scikit-learn’s encoders raise an error or map it to a default (the global mean, for
TargetEncoder); either way the model learns nothing about it. - Too many bins or knots. Each bin or knot adds a parameter fit on fewer rows, so variance rises. Neither bins nor splines extrapolate beyond the training range.
- Leakage across time. A cross-fitted target encoding still averages labels from the future when rows are ordered in time, the same failure as a shuffled split.
7.6 What the toy cannot show
The noise category is pure noise, and the curve is a single smooth function of one feature. Real categories carry some signal and some noise in unknown proportions, and real text needs far more than counts. The demonstrations show what each encoding can represent and how target encoding leaks, not which encoding is best for a given dataset.
7.7 Connections
- Cleaning: Missing Values, Duplicates, Outliers, Leakage: target encoding is the supervised preprocessing step that leaks, the second kind of leak measured there.
- Changing the Shape: splines and bins are another way to change what a linear model can fit in a feature.
- Sampling and Weighting: Part III turns from the representation lever to the sampling distribution and the weights.
- Choosing Data: text at scale, where duplicates and near-duplicates, not encodings, become the main data problem.