The Map

This book makes one argument in twelve chapters, and it helps to see the whole of it before starting. Three pictures carry it: the space in which every learning problem and every method sits, the lineage that connects the methods, and the handful of independent choices that every method name hides.

The space: what the feedback says, and who generated the data

Supervised learning and reinforcement learning are not two kinds of machine learning. They are regions of one space (What Makes a Problem RL). Along one axis, the feedback is instructive (a label says what the right answer was) or evaluative (a reward says only how good the given answer was), with comparisons and teacher signals in between. Along the other, the data is a fixed dataset or is generated by the learner’s own policy. A third axis, not shown as a position, is whether credit arrives immediately or must be assigned across a sequence of actions.

Figure 1: Every method in the book placed by the kind of feedback it learns from (vertical) and by who generated its training data (horizontal); marker shape shows whether credit is assigned per sequence or per token. Supervised fine-tuning and off-policy distillation sit in the instructive, fixed-data corner; the policy-gradient family sits in the evaluative, self-generated corner; offline preference methods sit between. Positions are read from the same records that generate the unified table.

The lineage: one tree, a few branches

Almost every named method modifies an earlier one, usually in exactly one respect. The tree grows from two main roots, the policy gradient (REINFORCE) and behavior cloning (SFT). From REINFORCE come PPO and PPO-based RLHF, which branches into the direct preference methods and GRPO’s line of critic-free policy gradients; the other critic-free methods (RLOO, ReMax) and the calibration reward modify REINFORCE directly. Filtered imitation and distillation grow from SFT. Two methods stand alone: best-of-\(n\) sampling, which modifies no training method, and TypeSafe’s RLCD, whose lineage is undisclosed.

Figure 2: The method lineage: each arrow points from a method to the one that modifies it, laid out by generation (the length of the chain of modifications from a root) and colored by family, with the year of each method’s primary source (SFT and best-of-\(n\), which have no single primary source, carry none). Generated with networkx from the same records as the unified table.

Each node links to a chapter:

Family	Methods	Chapter
classical RL	REINFORCE, PPO	Policy Gradients, Keeping Updates Stable
the hub	the KL-regularized objective, \(\pi^\star\), PPO-based RLHF	The Shared Target, and RLHF with PPO
direct preference	DPO, online DPO, IPO, KTO, SimPO, ORPO	Direct Preference Methods
critic-free policy gradients	RLOO, ReMax, GRPO, Dr. GRPO, DAPO, GSPO, CISPO	Policy Gradients Without a Critic
imitation and distillation	SFT, best-of-\(n\), rejection-sampling fine-tuning, STaR, ReST, RAFT, off- and on-policy distillation	Learning by Imitating a Better Distribution
calibration rewards	correctness + proper scoring rule; RLCD	Rewarding Calibration
where it is heading	multi-turn and agentic RL	What RL Actually Does to a Model

The axes a method name hides

A method name bundles several independent choices. The book unbundles them into the same fields for every method, its method card, and the unified table lines them up.

Axis	The choices	Where it is derived
Signal source	demonstrations · pairwise preferences · binary desirability · learned scalar reward · verifiable reward · AI or judge feedback · teacher log-probabilities · a scored probability	rewards, calibration
Data regime (\(q\))	offline · on-policy · slightly off-policy (batch reuse) · iterated	what makes a problem RL, stable updates
Estimator and weight (\(w\))	maximum likelihood · pairwise logistic likelihood · policy gradient with no baseline, a critic, leave-one-out, or group mean and std	policy gradients, critic-free
Stability	KL in the reward · KL in the loss · clip / trust region · reference-free · entropy bonus	stable updates
Credit granularity	sequence · token · step (process rewards)	the LLM as a policy
Models in memory	policy · reference · reward model · critic · teacher	the shared target

Reading paths

Front to back is the intended order: each part makes the next one feel inevitable. Three shorter paths also work. If you know classical RL and want the LLM-specific part, start at The LLM as a Policy. If you want the single idea that unifies the methods, read Policy Gradients and then The Shared Target. If you have a new paper in hand, go to the five-minute recipe in the decision guide.