Appendix C — Glossary

Each term links to the chapter that defines or uses it first. Symbols are in Notation.

Term	Meaning
Augmentation	Adding transformed copies of training examples; an assumption about how the transform acts on the label (Augmentation)
Back-transform bias	The bias from transforming a prediction made on a transformed scale (such as \(\log y\)) back to the original scale; also called retransformation bias (Changing the Shape)
Bayes risk	The lowest error any predictor can reach on a distribution: the floor of a learning curve (Learning Curves)
Class-conditional noise	Label noise whose flip probability depends only on the true class (Labels)
Complete-case analysis	Dropping every row with a missing value (Cleaning)
Concept shift	A change in \(p(y \mid x)\) between training and deployment, invisible to tests on inputs alone (Distribution Shift)
Confident learning	Finding likely label errors by comparing given labels with out-of-sample predicted probabilities (Labels)
Covariate shift	A change in \(p(x)\) with \(p(y \mid x)\) fixed (Distribution Shift)
Coverage-driven selection	Labeling only examples that cover cases not yet covered (Why Power Laws)
Cross-fitting	Computing a fitted feature for each row from a model that did not see that row (Encoding and Features)
Data card	The fixed-field record every technique gets: lever, assumption, effect, fit on, failure modes, alternatives, and the test that checks it (The Map)
Data-generating process	The population, the selection and the measurement that together produce a dataset (The Data-Generating Process)
Deduplication	Removing exact or near copies, within a training set or across a split (Choosing Data)
Density ratio	\(p(x)/q(x)\), the weight that corrects covariate shift (Sampling and Weighting)
Effective sample size	The number of unweighted examples a weighted sample is worth; it falls as weights grow uneven (Sampling and Weighting)
Entropy rate	The lowest cross-entropy per token any model of a source can reach (Scaling Laws)
Evaluation only	A technique that changes what is measured, not what is learned: splits, leakage control, shift detection (The Map)
Importance weighting	Weighting training examples by the density ratio to target \(p\) (Sampling and Weighting)
Invariance	A transform that leaves the label unchanged: \(p(y \mid T(x)) = p(y \mid x)\) (Augmentation)
Leakage	Information at training or evaluation time that will not be available at prediction time (Cleaning)
Learning curve	Expected test error as a function of the number of training examples (Learning Curves)
Lever	One of the four things a data technique can change: representation, sampling distribution \(q\), weights \(w\), labels (The Map)
Long tail	Many individually rare cases whose total probability is large (Why Power Laws)
MCAR, MAR, MNAR	Missing completely at random; at random given observed values; not at random (Cleaning)
Memorizing learner	A learner that is right on every case it has seen and wrong on every other: Hutter’s model (Why Power Laws)
Model collapse	Loss of a distribution’s tail when models are trained on their predecessors’ output (Choosing Data)
Prior shift	A change in the class prior \(p(y)\) with \(p(x \mid y)\) fixed (Sampling and Weighting)
Representation	How inputs and targets are encoded and transformed before the model sees them (The Map)
Sampling distribution \(q\)	The distribution training examples are drawn from (The Data-Generating Process)
Scaling law	An empirical power law relating loss to data, model size or compute (Scaling Laws)
Selection on \(x\), on \(y\)	A selection rule that depends only on the inputs leaves \(p(y \mid x)\) unchanged for a correctly specified model; one that depends on the target biases it (The Data-Generating Process)
Smearing	Correcting a back-transformed prediction by the mean of the back-transformed residuals (Changing the Shape)
SMOTE	Oversampling a minority class by interpolating between neighboring minority examples (Sampling and Weighting)
Stratified sampling	Sampling separately within known groups (The Data-Generating Process)
Target distribution \(p\)	The distribution the model will face at deployment (The Data-Generating Process)
Target encoding	Replacing a category by a statistic of the target within it (Encoding and Features)
Uncertainty sampling	Active learning that labels the example whose predicted probability is nearest 1/2 (Choosing Data)