Appendix D — Further Reading

Each entry was read for this book at the depth logged in the repository’s research-log.md; the note says what it is good for. Recent and contested results are in the chapters, with their dates.

D.1 Data collection, labels and cleaning

  • Gebru et al. (2018): the questions a dataset’s documentation should answer, from motivation to maintenance.
  • Çetinkaya-Rundel and Hardin (2024): an introductory statistics text with clear chapters on sampling designs and observational data.
  • Northcutt et al. (2019): confident learning, with its assumptions stated.
  • Buuren (2018): missing-data mechanisms and multiple imputation, in depth and free online.
  • Kuhn and Johnson (2019): a practical book on encodings, transformations and feature selection (read for its scope, not in full).

D.2 The training distribution

  • Saerens et al. (2002): adjusting a classifier’s probabilities to new class priors, and estimating those priors by EM.
  • Chawla et al. (2002): SMOTE, defined.
  • Moreno-Torres et al. (2012): definitions of covariate, prior-probability and concept shift.

D.3 How much data, and which

  • Viering and Loog (2021): a survey of learning-curve shapes, including the cases that are not power laws.
  • Hutter (2021): the exact model behind Why Power Laws.
  • Dohmatob et al. (2024): Hutter’s model trained on a distribution with its tail cut, the theory of model collapse on a long tail.
  • Michaud et al. (2023): the quantization model, linking power-law frequencies of skills to power-law scaling.
  • Hoffmann et al. (2022) and Muennighoff et al. (2023): compute-optimal allocation, and what repeated data is worth.
  • Sorscher et al. (2022): when to keep hard examples and when easy ones, derived and measured.
  • Settles (2009): a survey of active learning, from uncertainty sampling onward.
Buuren, Stef van. 2018. Flexible Imputation of Missing Data, Second Edition. Chapman; Hall/CRC. https://doi.org/10.1201/9780429492259.
Çetinkaya-Rundel, Mine, and Johanna Hardin. 2024. Introduction to Modern Statistics. Second. OpenIntro. https://openintro-ims.netlify.app/.
Chawla, N. V., K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. 2002. “SMOTE: Synthetic Minority Over-sampling Technique.” Journal of Artificial Intelligence Research 16: 321–57. https://doi.org/10.1613/jair.953.
Dohmatob, Elvis, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe. 2024. A Tale of Tails: Model Collapse as a Change of Scaling Laws. https://arxiv.org/abs/2402.07043.
Gebru, Timnit, Jamie Morgenstern, Briana Vecchione, et al. 2018. Datasheets for Datasets. https://arxiv.org/abs/1803.09010.
Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, et al. 2022. Training Compute-Optimal Large Language Models. https://arxiv.org/abs/2203.15556.
Hutter, Marcus. 2021. Learning Curve Theory. https://arxiv.org/abs/2102.04074.
Kuhn, Max, and Kjell Johnson. 2019. Feature Engineering and Selection. Chapman; Hall/CRC. https://doi.org/10.1201/9781315108230.
Michaud, Eric J., Ziming Liu, Uzay Girit, and Max Tegmark. 2023. The Quantization Model of Neural Scaling. https://arxiv.org/abs/2303.13506.
Moreno-Torres, Jose G., Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V. Chawla, and Francisco Herrera. 2012. “A unifying view on dataset shift in classification.” Pattern Recognition 45 (1): 521–30. https://doi.org/10.1016/j.patcog.2011.06.019.
Muennighoff, Niklas, Alexander M. Rush, Boaz Barak, et al. 2023. Scaling Data-Constrained Language Models. https://arxiv.org/abs/2305.16264.
Northcutt, Curtis G., Lu Jiang, and Isaac L. Chuang. 2019. Confident Learning: Estimating Uncertainty in Dataset Labels. https://arxiv.org/abs/1911.00068.
Saerens, Marco, Patrice Latinne, and Christine Decaestecker. 2002. “Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure.” Neural Computation 14 (1): 21–41. https://doi.org/10.1162/089976602753284446.
Settles, Burr. 2009. Active Learning Literature Survey. Computer Sciences Technical Report 1648. University of Wisconsin–Madison. https://burrsettles.com/pub/settles.activelearning.pdf.
Sorscher, Ben, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. 2022. Beyond neural scaling laws: beating power law scaling via data pruning. https://arxiv.org/abs/2206.14486.
Viering, Tom, and Marco Loog. 2021. The Shape of Learning Curves: a Review. https://arxiv.org/abs/2103.10948.