Every chapter uses the symbols on this page rather than defining its own. When a chapter needs a new symbol, it is added here in the same change. If two chapters ever use one letter for two things, that is a bug in the book.
This book shares conventions with its siblings where the topics overlap: loss-functions-lab for losses, risks and likelihoods, and rl-for-llms for the sampling distribution \(q\) and the weight \(w\). The section on resolved conflicts below lists every place where this book departs from them, and why.
Data and distributions
| \(x \in \mathcal{X}\) |
an input (feature vector); \(x_j\) its \(j\)-th feature |
| \(y \in \mathcal{Y}\) |
a target: a class, a real value, a token |
| \(d\) |
the input dimension |
| \(n\) |
the number of training examples (the sample size) |
| \(D = \{(x_i, y_i)\}_{i=1}^n\) |
a dataset; \(D_{\text{train}}\) and \(D_{\text{test}}\) when both appear |
| \(X\) |
the \(n \times d\) design matrix |
| \(p\) |
the target distribution: the one the model will face. \(p(x, y)\), \(p(x)\), \(p(y \mid x)\) |
| \(q\) |
the training (sampling) distribution: the one examples are drawn from. Same arguments as \(p\) |
| \(w\) |
a per-example weight; under covariate shift, \(w(x) = p(x)/q(x)\), the density ratio |
| \(p(y)\), \(q(y)\) |
class priors under the target and training distributions |
Learning
| \(\theta\) |
model parameters |
| \(f_\theta\), \(\hat f\) |
a model; a fitted model |
| \(\ell(y, \hat y)\) |
the per-example loss |
| \(R(\theta) = \mathbb{E}_{(x,y)\sim p}[\ell]\) |
the risk under the target distribution |
| \(\hat R(\theta)\) |
the empirical risk on a sample |
| \(R^\star\) |
the Bayes risk: the lowest risk any predictor can reach (the floor of a learning curve) |
| \(E_n\) |
the expected test error after \(n\) training examples: the learning curve |
| \(\beta\) |
the exponent of a power-law learning curve, \(E_n \propto n^{-\beta}\) |
Collection, labels and cleaning (Part I and II)
| \(s(x, y)\) |
the probability that an example drawn from \(p\) is kept when the data is collected; then \(q \propto s\,p\) |
| \(W_h\), \(S_h^2\), \(S^2\) |
in stratified sampling: stratum \(h\)’s share of the population, its variance, the overall variance |
| \(\eta(x)\) |
the clean posterior \(P(y = 1 \mid x)\) under \(p\) |
| \(\tilde y\) |
a recorded (possibly wrong) label, as opposed to the true \(y\) |
| \(\rho_0\), \(\rho_1\) |
class-conditional flip rates: \(P(\tilde y = 1 \mid y = 0)\) and \(P(\tilde y = 0 \mid y = 1)\) |
| \(t_j\) |
in confident learning, the average predicted probability of class \(j\) among examples labeled \(j\) |
| \(r\) |
the share of a column’s values that is missing |
| \(m(x)\) |
the conditional mean of the transformed target, \(\mathbb{E}[\log y \mid x]\) in Changing the Shape |
| \(\hat\varepsilon_i\) |
a training residual of a fitted model |
The long-tail testbed (T2)
| \(p_i\) |
the probability of feature \(i\) under the target distribution. Hutter (2021) writes \(\theta_i\) |
| \(\alpha\) |
the Zipf exponent: \(p_i \propto i^{-(\alpha+1)}\), as in Hutter (2021). Dohmatob et al. (2024) write \(p_i \propto i^{-\beta}\), so their \(\beta\) is this book’s \(\alpha + 1\) |
| \(K\) |
the support size, when the feature set is truncated for computation |
| \(E^{\text{oracle}}_n\) |
the error of the oracle coverage selector that labels features \(1, \dots, n\): \(\sum_{i>n} p_i\) |
| \(M\) |
the size of an unlabeled pool a selector may inspect |
| \(m\), \(D_m\) |
the number of draws from a uniform stream, and the number of distinct features among them (the labels used when only new features are labeled) |
| \(s\), \(A\) |
\(s = \alpha + 1\) and \(A = 1/\zeta(s)\), so that \(p_i = A\,i^{-s}\) |
| \(c\) |
the constant in an asymptotic power law \(E_n \approx c\,n^{-\beta}\) |
| \(\zeta(s)\), \(\Gamma\) |
the Riemann zeta function \(\sum_{i\ge1} i^{-s}\); the gamma function |
The other testbeds
| \(z\), \(\Sigma\) |
T1’s latent Gaussian features and their covariance; observed \(x = z\) or \(x = e^{z}\) |
| \(a\), \(b\) |
T1’s regression coefficients: \(\log y = a + b^\top z + \varepsilon\) |
| \(u\), \(u_0\) |
T1’s classification coefficients: \(P(y = 1 \mid z) = \sigma(u^\top z + u_0)\) |
| \(\varepsilon\) |
a noise term, with standard deviation \(\sigma_\varepsilon\) |
| \(\phi\) |
T3: the angle between student and teacher; the test error is \(\phi/\pi\) |
Resolved conflicts
- Class priors are \(p(y)\) and \(q(y)\), not \(\pi_y\). The sibling books reserve \(\pi\) for policies. The lens of this book also makes the prior-shift correction read as what it is: a change from \(q\) to \(p\). SPEC v0.1 §8 claim 10 wrote \(\pi_y\); that claim is restated in this notation.
- Hutter’s \(\theta_i\) is \(p_i\) here. \(\theta\) means model parameters in every sibling book, and in T2 the feature distribution is the target distribution \(p\), which makes the comparison of \(q\) with \(p\) direct (as in Dohmatob et al. 2024, who also write \(p_i\) and \(q_i\)).
- Scaling Laws keeps the papers’ symbols (\(D\) tokens, \(N\) parameters; \(E\), \(A\), \(B\), \(\alpha\), \(\beta\) as fitted by Hoffmann et al.; \(\alpha_D\), \(\alpha_N\) as in Kaplan et al.), so readers can match them to the sources; the chapter says so where they first appear.
- \(\sigma\) is the sigmoid, as in the siblings, so noise scale is \(\sigma_\varepsilon\).
- \(\beta\) is a learning-curve exponent here. In rl-for-llms, \(\beta\) is the KL strength; the two books never need both.
Recurring visual devices
Data cards (callout-tip): every technique, generated from scripts/technique_data.py, with the same eight fields in the same order: changes, assumption, what it does to the learned function, which models care, fit on, failure modes, alternatives, checked by. The decision guide is assembled from them.
Lever colors, in every figure: representation blue, sampling distribution \(q\) orange, weights \(w\) aqua, labels violet. Exact ground truth is drawn in black.