12 Decision Guide and the Unified Table
The book has derived some twenty methods, and every one of them turned out to be a choice along the same few axes: where the training signal comes from, which samples the gradient uses, what weight each sample gets, what keeps the update stable, at what granularity credit is assigned, and which target the whole procedure is aiming at. This chapter turns that into two practical tools. The first is a procedure for choosing a method given the signal and the budget you actually have, and for knowing what to watch while it trains. The second is a way to place a new method on the map in a few minutes, by asking the same questions of its paper. The unified table, assembled from the method cards, is the reference for both.
12.1 A procedure for choosing a method
Work through five questions in order. Each one narrows the families that fit.
1. What signal do you have?
| You have | The natural families | Chapter |
|---|---|---|
| demonstrations of good behavior | SFT (behavior cloning); off-policy distillation if they come from a model | LLM as a policy, imitation |
| pairwise preferences, already collected | DPO and its variants; or a reward model + RL | direct preference, rewards |
| a preference annotator you can query (human or model) | online DPO; reward model + on-policy RL | direct preference, shared target |
| thumbs-up / thumbs-down on single responses | KTO; or filtering | direct preference |
| a verifier (tests, answer checker) | critic-free policy gradients (RLOO, GRPO and its fixes); rejection-sampling fine-tuning | critic-free, imitation |
| a stronger model whose log-probabilities you can query | on-policy distillation | imitation |
| outcomes against which a stated probability can be scored | correctness + a bounded proper scoring rule | calibration |
2. Can you sample from the current policy and score the samples during training? If yes, on-policy methods are available, and they are the ones that learn from the model’s own mistakes (the DAgger argument of the LLM-as-policy chapter) and that, on the toy language, kept converging where offline methods drifted. If no, you are offline: DPO-family methods on a fixed dataset, or one round of filtering. Expect the offline costs measured in the direct-preference chapter: no exploration, overfitting the dataset’s deterministic preferences, and the policy drifting away from the data’s distribution. A middle path is iterated: collect, train, re-collect.
3. What can you hold in memory? PPO-based RLHF needs four models (policy, reference, reward model, critic). Critic-free methods with a verifiable reward need two (policy, reference) or one if the KL term is dropped. DPO needs two, and the reference’s log-probabilities can be precomputed. SimPO and ORPO need one. On-policy distillation needs the student and the teacher.
4. What target do you want, and does your implementation actually aim at it? If the goal is the KL-regularized optimum \(\pi^\star\), check how the KL is implemented: “\(k_1\) in the reward” and “\(k_2\) as a loss” target it; “\(k_3\) as a loss” regularizes the forward KL instead and lands elsewhere (keeping updates stable). Check what the advantage normalization does: dividing by a group’s standard deviation reweights prompts and weakens the KL relative to the reward, and dividing by each response’s length biases length (critic-free). If the method has no reference model at all (SimPO, ORPO, DAPO), it has no \(\pi^\star\); decide whether that matters for your use.
5. What will you monitor? Every failure mode in the book has a signature that can be watched.
| Watch | Because | Chapter |
|---|---|---|
| KL from \(\pi_{\text{ref}}\) | the axis along which overoptimization happens; a KL penalty mostly decides where on it training stops | rewards |
| reward on a held-out true metric, not just the training reward | a learned or hackable reward keeps rising after the true one peaks | rewards |
| response length (for correct and incorrect responses separately) | length bias from reward models and from per-response normalization | rewards, critic-free |
| entropy, and the diversity of correct answers | collapse from unregularized optimization and iterated filtering | imitation, what RL does |
| fraction of groups with zero reward variance | the share of the batch that carries no signal | critic-free |
| fraction of tokens clipped | clipping silences tokens rather than flagging problems | stable updates |
| likelihood of the preferred responses (preference methods) | likelihood displacement | direct preference |
| calibration of any stated confidence, per data slice | pooled calibration can hide per-slice base-rate errors | calibration |
12.2 Classical RL vs LLM-era RL, revisited
The LLM-as-policy chapter gave a table of what became of each classical concept. Having derived the methods, three of its rows can be stated more sharply.
- The critic was dropped for single-turn reasoning because a group of samples gives an unbiased per-prompt baseline at the cost of extra generation, and the measured benefit of an exact per-token critic over a per-prompt baseline was small on the toy language (policy gradients). It returns where consequences are delayed across turns (what RL does).
- The KL to a reference is the one genuinely new structural element, and its implementation decides the target (stable updates, shared target).
- Off-policy learning survives mainly as offline preference optimization (DPO), which avoids importance sampling altogether by not sampling, and as mild staleness in asynchronous pipelines, corrected by ratios and clips.
12.3 The unified table
One row per method, with the fields that matter most for choosing between them. The full table, with every method-card field and a filter box, is on its own page.
| Method | Family | Signal | Samples from (\(q\)) | Per-sample weight (\(w\)) | Targets | What changed vs its parent |
|---|---|---|---|---|---|---|
| REINFORCE | classical RL | scalar reward | \(\pi_\theta\) (on-policy) | \(r\) or \(r-b\) | \(\max\mathbb{E}[r]\) (no KL) | (root) score-function gradient |
| SFT (behavior cloning) | imitation / distillation | demonstrations | the dataset | 1 | forward KL to the data | (root) maximum likelihood on demonstrations |
| PPO (classical) | classical RL | environment reward per step | \(\pi_{\text{old}}\), reused for \(K\) epochs | \(\rho_t\hat A_t\), zeroed where the clip binds | expected return | clipped trust region, critic + GAE, minibatch reuse |
| PPO-RLHF | policy gradient | reward model at the end, \(-\beta\log(\pi/\pi_{\text{ref}})\) per token | \(\pi_{\text{old}}\) (fresh rollouts, few epochs) | \(\rho_t\hat A_t\) (GAE), clipped | \(\pi^\star\) of the learned reward | learned reward model; per-token KL to a fixed \(\pi_{\text{ref}}\) in the reward |
| RLOO | policy gradient | scalar reward (KL folded in) | \(\pi_\theta\), \(G\) per prompt | \(r_i - \mathrm{mean}_{j\ne i} r_j\) | \(\pi^\star\) | leave-one-out baseline from G samples; no critic |
| ReMax | policy gradient | scalar reward | \(\pi_\theta\) + one greedy decode | \(r(y) - r(\bar y_{\text{greedy}})\) | \(\pi^\star\) | greedy-response reward as baseline |
| GRPO | policy gradient | scalar reward (often verifiable) | \(\pi_{\text{old}}\), \(G\) per prompt | \((r_i-\bar r)/\mathrm{std}\), clipped ratio, \(1/\lvert o_i\rvert\) | approx. \(\pi^\star\) (forward-KL regularizer) | critic replaced by group mean/std; KL moved to the loss (\(k_3\)) |
| Dr. GRPO | policy gradient | scalar reward | \(\pi_{\text{old}}\), \(G\) per prompt | \(r_i - \bar r\), clipped ratio, constant normalizer | reward objective without length/difficulty reweighting | removes the std and \(1/\lvert o_i\rvert\) normalizations |
| DAPO | policy gradient | verifiable reward + overlong shaping | \(\pi_{\text{old}}\), groups with non-zero variance only | group-normalized advantage, \(\epsilon_{\text{high}}>\epsilon_{\text{low}}\), token-level sum | reward objective (no KL anchor) | clip-higher, dynamic sampling, token-level loss, overlong shaping, no KL |
| GSPO | policy gradient | scalar reward | \(\pi_{\text{old}}\), \(G\) per prompt | group-normalized advantage, clipped \(s_i(\theta)\) | reward objective, sequence-level trust region | sequence-level importance ratio and clip |
| CISPO | policy gradient | scalar reward | \(\pi_{\text{old}}\) | \(\mathrm{sg}(\mathrm{clip}(\rho_t))\hat A\) | reward objective | clips the importance weight, never drops token gradients |
| DPO | preference | pairwise preferences | fixed dataset of pairs | \(\pm\beta\sigma(\hat r^- - \hat r^+)\) | \(\pi^\star\) (infinite full-support data) | closed form inverted: reward expressed through \(\pi\); no RM, no sampling |
| Online DPO | preference | pairwise preferences (online annotator) | \(\pi_\theta\) (pairs) | \(\pm\beta\sigma(\hat r^- - \hat r^+)\) | \(\pi^\star\) | pairs sampled from the current policy and labelled on the fly |
| IPO | preference | pairwise preferences | fixed dataset of pairs | \(\pm 2(\frac{1}{2\tau} - h_\theta)\) | \(\pi_{\text{ref}}\) tilted by preference probability | bounded (identity) preference transform: squared loss to a fixed target |
| KTO | preference | binary labels on single responses | fixed dataset | \(\pm c_{D/U}\beta\sigma'(\cdot)\) around \(z_0\) | prospect-theoretic utility of the implicit reward | unpaired binary desirable/undesirable labels; reference point |
| SimPO | preference | pairwise preferences | fixed dataset of pairs | \(\pm\frac{\beta}{\lvert y\rvert}\sigma(-\text{margin})\) | no \(\pi^\star\) (no reference) | reference-free, length-normalized reward, target margin |
| ORPO | preference | preferences + preferred responses | fixed dataset | \(+1\) (SFT) plus signed odds-ratio weight | SFT objective with a preference penalty | SFT + odds-ratio preference term; no reference; single stage |
| Best-of-n | imitation / distillation | reward model or verifier | \(\pi_{\text{ref}}\), \(n\) samples | 1 on the argmax, 0 otherwise | a distribution just below the \(\pi^\star_\beta\) frontier | (inference-time) pick the highest-scoring of n samples |
| Rejection-sampling FT / STaR / ReST / RAFT | imitation / distillation | reward used as a filter | \(\pi_{\text{ref}}\) or the current policy | \(\mathbb{1}[\text{good}]\) (or \(e^{r/\beta}\)) | \(\pi^\star\) with exact acceptance; sharper (collapsing) when iterated with hard filters | SFT on the policy’s own filtered samples, iterated |
| Off-policy distillation | imitation / distillation | teacher samples | the teacher \(\pi_T\) | 1 | forward KL to the teacher | demonstrations come from a teacher model |
| On-policy distillation / GKD | imitation / distillation | teacher per-token log-probs | the student \(\pi_\theta\) | \(-(\log\pi_\theta - \log\pi_T)\) per token | the teacher (reverse KL) | student samples, teacher scores every token (dense reward) |
| Correctness + proper scoring rule (book’s reconstruction) | calibration | \(\mathbb{1}[\text{correct}] - (p - \mathbb{1}[\text{correct}])^2\) | \(\pi_\theta\), several per prompt | reward minus leave-one-out baseline | best answer + honest probability | reward scores the stated confidence with a proper scoring rule |
| RLCD (TypeSafe AI) | calibration | calibrated decisions (objective public; reward undisclosed) | undisclosed | undisclosed | calibrated probabilities over typed answers (stated goal) | undisclosed |
12.4 Placing a new paper on the map in five minutes
When a new post-training method appears, ask its paper the method-card questions, in this order. Most papers answer each in a sentence, usually in the method section or the main equation.
- Signal. What does the training signal say, and who produced it: demonstrations, preferences, binary labels, a learned reward, a verifier, a judge, a teacher’s log-probabilities, a scored probability?
- Sampling distribution \(q\). Which responses does the gradient use: a fixed dataset, the current policy, an older policy, a teacher?
- Weight \(w\). Write the gradient as \(\mathbb{E}_{q}[w\,\nabla\log\pi_\theta]\). What is \(w\), per sample or per token? Can it be negative? Is it normalized, and by what?
- Stability. What stops the policy moving too far: a KL (to what, in which direction, implemented how), a clip (on what ratio, at which granularity), nothing?
- Credit. Is the weight per sequence, per step or per token?
- Target. If everything were exact, which distribution would the procedure converge to? Is it \(\pi^\star\) for some reward and \(\beta\), or something else?
Then name the classical idea it reuses, and the method it modifies. That is a row of the unified table.
A worked example: Math-Shepherd. The paper trains a process reward model whose label for each reasoning step is the fraction of \(N\) completions from that step that reach the correct answer, and then “employ[s it] to reinforce LLMs with step-by-step Proximal Policy Optimization” (Wang et al. 2023). Signal: a learned process reward, per step. Sampling: the current policy, as in PPO. Weight: PPO’s clipped advantage, now with rewards at every step instead of only at the end. Stability: PPO’s clip (and whatever KL the implementation uses, which the reader should check). Credit: step level, from a reward that is a Monte Carlo estimate of the value of each partial solution. Target: the KL-regularized optimum of the process-reward-shaped objective. Classical lineage: PPO with a learned value function used as a shaping reward. Parent: PPO-based RLHF, with the outcome reward model replaced by a process reward model. The rewards chapter predicts its risk: a learned value is a proxy, and a policy optimized against it will find where it is wrong.
12.5 The final test
The book set out to move a reader from
“RLHF, DPO, and GRPO are three alignment techniques I’ve heard of.”
to
“All three target a KL-regularized reward objective whose optimum is the reference policy tilted by \(\exp(r/\beta)\). RLHF learns a reward from preferences and climbs toward that optimum with an on-policy policy gradient, using a learned critic as baseline and a clip as trust region. DPO uses the closed form to cancel the reward model out and fits the policy to fixed preference pairs by logistic regression, giving up exploration. GRPO keeps the on-policy policy gradient but replaces the critic with a Monte Carlo baseline computed from a group of samples per prompt, which works well when rewards are verifiable.”
with one amendment that the toy language forced: GRPO as published, with its KL estimator in the loss and its standard-deviation normalization, regularizes a different divergence than RLHF and lands somewhere else. And from
“RL is a different kind of ML from supervised learning.”
to
“RL differs in where the data comes from and what the feedback says. Its gradient is a weighted maximum-likelihood gradient on samples the learner chose to generate. SFT is the special case of weight 1 on someone else’s samples.”
12.6 Connections
- The Map draws the lineage this chapter tabulates, and the space of feedback, data and credit in which every row of the unified table sits.
- Policy Gradients introduced the \((q, w)\) form that question 2 and 3 of the five-minute recipe ask about.
- The Shared Target, and RLHF with PPO supplies the target that question 6 asks about, and the signature figure that compares every route to it.
- The terminology traps and acronym decoder appendices collect the vocabulary problems that made the methods look more different than they are.