What the Model Sees

Training Data from First Principles

Where data for learning comes from, how it is transformed, how the training distribution is shaped, and how much of it (and which of it) a model needs, with every claim checked against synthetic data whose ground truth is known exactly.

Welcome

A model learns the distribution it is trained on, through the representation it is given. This book is about that sentence. It asks where data for learning comes from, how it is transformed, how the training distribution is shaped, and how much of it (and which of it) a model needs.

Every technique gets a data card that says which of four things it changes: the representation, the sampling distribution, the per-example weights, or the labels. Every checkable statement is a test in the repository, and every figure is computed from synthetic data whose ground truth is known exactly.

Start at the map.

How this book was made

The book was written by Claude (Anthropic), working through Claude Code, with Ioannis Antoniadis. Claude did the research, drafting, code, tests and figures. Ioannis Antoniadis set the direction, decided every question reserved for the owner, reviewed the chapters, and is responsible for every claim. The model used throughout was Claude Opus 5.5.

The process exists because a model writing from memory is most wrong about exactly the details a reader copies: equations, library defaults, reported numbers and dates. So every technical statement was taken from a source that was opened and read, and the depth of each reading is logged in the repository’s research-log.md. Every checkable statement is a test, from Sampling and Weighting on, every number in the text is written by the script that computed it (earlier numbers were copied from fresh runs and rechecked by the final audit), and every figure was looked at after it was generated. Independent reviews by fresh sessions, with no drafting context, are kept in the repository’s audits/ folder and review.md.