13 The Unified Table
Every method in the book on one page, one row per method, with the columns of the method cards plus what each method changed relative to the one it modifies. The rows are generated from a single data file (scripts/method_data.py), transcribed from the method cards in the chapters, so the table and the chapters say the same thing. Each method’s name links to the chapter where its card is derived. “Undisclosed” marks fields of RLCD that TypeSafe AI has not published.
Type in the box to filter rows (for example “verifiable”, “offline”, “critic”, “token”).
| Method | Family | Signal | Samples from (\(q\)) | Per-sample weight (\(w\)) | Stability | Credit granularity | Models in memory | Targets | Classical lineage | What changed vs its parent |
|---|---|---|---|---|---|---|---|---|---|---|
| REINFORCE | classical RL | scalar reward | \(\pi_\theta\) (on-policy) | \(r\) or \(r-b\) | small learning rate | sequence (reward-to-go per step in MDPs) | policy | \(\max\mathbb{E}[r]\) (no KL) | Williams 1992 | (root) score-function gradient |
| SFT (behavior cloning) | imitation / distillation | demonstrations | the dataset | 1 | none needed | token (teacher forcing) | policy | forward KL to the data | behavior cloning; DAgger’s problem case | (root) maximum likelihood on demonstrations |
| PPO (classical) | classical RL | environment reward per step | \(\pi_{\text{old}}\), reused for \(K\) epochs | \(\rho_t\hat A_t\), zeroed where the clip binds | ratio clip; entropy bonus; optional KL to \(\pi_{\text{old}}\) | per step (GAE) | policy, critic | expected return | TRPO; actor-critic | clipped trust region, critic + GAE, minibatch reuse |
| PPO-RLHF | policy gradient | reward model at the end, \(-\beta\log(\pi/\pi_{\text{ref}})\) per token | \(\pi_{\text{old}}\) (fresh rollouts, few epochs) | \(\rho_t\hat A_t\) (GAE), clipped | KL to \(\pi_{\text{ref}}\) in the reward; clip | token (critic + GAE) | policy, reference, reward model, critic | \(\pi^\star\) of the learned reward | PPO actor-critic on the token MDP | learned reward model; per-token KL to a fixed \(\pi_{\text{ref}}\) in the reward |
| RLOO | policy gradient | scalar reward (KL folded in) | \(\pi_\theta\), \(G\) per prompt | \(r_i - \mathrm{mean}_{j\ne i} r_j\) | KL-shaped reward | sequence | policy, reference, reward | \(\pi^\star\) | REINFORCE + Monte Carlo baseline (Kool et al. 2019) | leave-one-out baseline from G samples; no critic |
| ReMax | policy gradient | scalar reward | \(\pi_\theta\) + one greedy decode | \(r(y) - r(\bar y_{\text{greedy}})\) | KL to \(\pi_{\text{ref}}\) in the reward | sequence | policy, reference, reward | \(\pi^\star\) | self-critical sequence training | greedy-response reward as baseline |
| GRPO | policy gradient | scalar reward (often verifiable) | \(\pi_{\text{old}}\), \(G\) per prompt | \((r_i-\bar r)/\mathrm{std}\), clipped ratio, \(1/\lvert o_i\rvert\) | clip; \(k_3\) KL in the loss | sequence advantage on every token | policy, reference | approx. \(\pi^\star\) (forward-KL regularizer) | REINFORCE + MC baseline + PPO trust region | critic replaced by group mean/std; KL moved to the loss (\(k_3\)) |
| Dr. GRPO | policy gradient | scalar reward | \(\pi_{\text{old}}\), \(G\) per prompt | \(r_i - \bar r\), clipped ratio, constant normalizer | clip | sequence | policy (+ reference) | reward objective without length/difficulty reweighting | REINFORCE + MC baseline + clip | removes the std and \(1/\lvert o_i\rvert\) normalizations |
| DAPO | policy gradient | verifiable reward + overlong shaping | \(\pi_{\text{old}}\), groups with non-zero variance only | group-normalized advantage, \(\epsilon_{\text{high}}>\epsilon_{\text{low}}\), token-level sum | asymmetric clip; no KL | sequence advantage, token aggregation | policy | reward objective (no KL anchor) | REINFORCE + MC baseline + asymmetric trust region | clip-higher, dynamic sampling, token-level loss, overlong shaping, no KL |
| GSPO | policy gradient | scalar reward | \(\pi_{\text{old}}\), \(G\) per prompt | group-normalized advantage, clipped \(s_i(\theta)\) | sequence-level clip | sequence throughout | policy (+ reference) | reward objective, sequence-level trust region | importance-sampled PG on the whole action | sequence-level importance ratio and clip |
| CISPO | policy gradient | scalar reward | \(\pi_{\text{old}}\) | \(\mathrm{sg}(\mathrm{clip}(\rho_t))\hat A\) | clipped IS weight; token-level aggregation | sequence advantage, token weights | policy (+ reference) | reward objective | truncated importance sampling (IMPALA’s V-trace) | clips the importance weight, never drops token gradients |
| DPO | preference | pairwise preferences | fixed dataset of pairs | \(\pm\beta\sigma(\hat r^- - \hat r^+)\) | log-ratio to \(\pi_{\text{ref}}\) in the loss | sequence | policy, reference | \(\pi^\star\) (infinite full-support data) | none in RL: Bradley-Terry logistic regression | closed form inverted: reward expressed through \(\pi\); no RM, no sampling |
| Online DPO | preference | pairwise preferences (online annotator) | \(\pi_\theta\) (pairs) | \(\pm\beta\sigma(\hat r^- - \hat r^+)\) | log-ratio to \(\pi_{\text{ref}}\) | sequence | policy, reference, annotator | \(\pi^\star\) | DPO’s loss with RL’s data regime | pairs sampled from the current policy and labelled on the fly |
| IPO | preference | pairwise preferences | fixed dataset of pairs | \(\pm 2(\frac{1}{2\tau} - h_\theta)\) | bounded target | sequence | policy, reference | \(\pi_{\text{ref}}\) tilted by preference probability | none in RL: least squares | bounded (identity) preference transform: squared loss to a fixed target |
| KTO | preference | binary labels on single responses | fixed dataset | \(\pm c_{D/U}\beta\sigma'(\cdot)\) around \(z_0\) | log-ratio to \(\pi_{\text{ref}}\); \(z_0\) | sequence | policy, reference | prospect-theoretic utility of the implicit reward | none in RL: prospect theory | unpaired binary desirable/undesirable labels; reference point |
| SimPO | preference | pairwise preferences | fixed dataset of pairs | \(\pm\frac{\beta}{\lvert y\rvert}\sigma(-\text{margin})\) | margin \(m\); no KL | sequence, length-normalized | policy | no \(\pi^\star\) (no reference) | none in RL: margin logistic loss | reference-free, length-normalized reward, target margin |
| ORPO | preference | preferences + preferred responses | fixed dataset | \(+1\) (SFT) plus signed odds-ratio weight | SFT term; no KL | sequence | policy | SFT objective with a preference penalty | behavior cloning + pairwise loss | SFT + odds-ratio preference term; no reference; single stage |
| Best-of-n | imitation / distillation | reward model or verifier | \(\pi_{\text{ref}}\), \(n\) samples | 1 on the argmax, 0 otherwise | none (no training) | sequence | policy, reward | a distribution just below the \(\pi^\star_\beta\) frontier | rejection / order-statistic sampling | (inference-time) pick the highest-scoring of n samples |
| Rejection-sampling FT / STaR / ReST / RAFT | imitation / distillation | reward used as a filter | \(\pi_{\text{ref}}\) or the current policy | \(\mathbb{1}[\text{good}]\) (or \(e^{r/\beta}\)) | none needed (supervised) | sequence | policy (+ filter) | \(\pi^\star\) with exact acceptance; sharper (collapsing) when iterated with hard filters | reward-weighted regression; expert iteration | SFT on the policy’s own filtered samples, iterated |
| Off-policy distillation | imitation / distillation | teacher samples | the teacher \(\pi_T\) | 1 | none needed | token (teacher forcing) | student (+ teacher to sample) | forward KL to the teacher | behavior cloning of the teacher | demonstrations come from a teacher model |
| On-policy distillation / GKD | imitation / distillation | teacher per-token log-probs | the student \(\pi_\theta\) | \(-(\log\pi_\theta - \log\pi_T)\) per token | dense signal | token | student, teacher | the teacher (reverse KL) | DAgger; PG with a dense shaped reward | student samples, teacher scores every token (dense reward) |
| Correctness + proper scoring rule (book’s reconstruction) | calibration | \(\mathbb{1}[\text{correct}] - (p - \mathbb{1}[\text{correct}])^2\) | \(\pi_\theta\), several per prompt | reward minus leave-one-out baseline | small learning rate (KL optional) | one decision (contextual bandit) | policy (+ verifier) | best answer + honest probability | REINFORCE in a contextual bandit; forecasting scoring rules | reward scores the stated confidence with a proper scoring rule |
| RLCD (TypeSafe AI) | calibration | calibrated decisions (objective public; reward undisclosed) | undisclosed | undisclosed | undisclosed | undisclosed | undisclosed | calibrated probabilities over typed answers (stated goal) | undisclosed | undisclosed |
Two reading notes. A method “targets \(\pi^\star\)” only in the idealized sense of the shared-target chapter: with enough data, a policy that can represent it, and the KL implemented as in that chapter. And “fully on-policy” methods are, in practice, slightly off-policy wherever they reuse a batch for several updates.