Appendix C — Acronym Decoder
Much of the confusion this book addresses is terminological: a stream of acronyms, each naming a bundle of choices. Each entry below gives the expansion, a one-line description in terms of the book’s axes (what it samples, how it weights, what it targets), and the chapter where it is derived.
| Acronym | Expansion | What it is, from first principles | Chapter |
|---|---|---|---|
| AWR | Advantage-Weighted Regression | supervised learning on logged actions weighted by \(\exp(A/\beta)\); fits the KL-constrained optimum | 9 |
| BoN | Best-of-\(n\) | sample \(n\), keep the highest-scoring; inference-time filtering | 9, 5 |
| BT | Bradley–Terry | \(P(y^+\succ y^-) = \sigma(r(y^+) - r(y^-))\); turns comparisons into a reward | 5 |
| CAI | Constitutional AI | AI-generated critiques and preferences from a list of principles, then RL from AI feedback | 5 |
| CISPO | Clipped Importance Sampling Weight Policy Optimization | policy gradient whose importance weight is clipped but never zeroed | 8 |
| DAgger | Dataset Aggregation | imitation learning that collects expert labels on the learner’s own states | 1, 4 |
| DAPO | Decoupled Clip and Dynamic sAmpling Policy Optimization | GRPO with asymmetric clip, zero-variance groups dropped, token-level loss, no KL | 8 |
| DPO | Direct Preference Optimization | logistic regression of preference pairs on the implicit reward \(\beta\log(\pi/\pi_{\text{ref}})\); targets \(\pi^\star\) offline | 7 |
| Dr. GRPO | Group Relative Policy Optimization Done Right | GRPO without the std and per-response length normalizations | 8 |
| ECE | Expected Calibration Error | average gap between stated confidence and observed accuracy | 10 |
| GAE | Generalized Advantage Estimation | exponentially weighted sum of TD errors; the bias–variance knob \(\lambda\) | 2 |
| GKD | Generalized Knowledge Distillation | distillation on a mix of fixed data and the student’s own samples | 9 |
| GRPO | Group Relative Policy Optimization | REINFORCE with a group-normalized Monte Carlo baseline, PPO clip, \(k_3\) KL in the loss | 8 |
| GSPO | Group Sequence Policy Optimization | GRPO with a sequence-level importance ratio and clip | 8 |
| IPO | Identity Preference Optimization | DPO with a bounded (squared) loss; targets the preference-probability tilt | 7 |
| KL | Kullback–Leibler divergence | reverse \(D_{\mathrm{KL}}(\pi\|\pi_{\text{ref}})\) in RL objectives; forward in SFT | 3 |
| KTO | Kahneman–Tversky Optimization | unpaired binary labels scored against a reference point | 7 |
| LLM-as-judge | — | a general model prompted to compare or grade, used as a reward | 5 |
| MDP | Markov Decision Process | states, actions, transitions, rewards; for text: prefix, token, append, terminal reward | 1, 4 |
| ORM / PRM | Outcome / Process Reward Model | a reward for the final answer / for each step; a PRM trained on rollouts estimates a value function | 5 |
| ORPO | Odds Ratio Preference Optimization | SFT plus an odds-ratio preference term; no reference model | 7 |
| PPO | Proximal Policy Optimization | policy gradient with a clipped importance ratio (a soft trust region) and a critic | 3, 6 |
| PPO-ptx | PPO with pretraining mix | InstructGPT’s PPO plus a pretraining-likelihood term | 6 |
| RAFT | Reward rAnked FineTuning | fine-tune on the reward-ranked best of the policy’s samples | 9 |
| ReMax | — | REINFORCE with the greedy response’s reward as baseline | 8 |
| ReST | Reinforced Self-Training | grow (sample) / improve (filter and fine-tune), iterated | 9 |
| RFT | Rejection-sampling Fine-Tuning | SFT on the policy’s filtered correct samples | 9 |
| RLAIF | RL from AI Feedback | RLHF with preference labels from a model | 5 |
| RLCD (2026) | Reinforcement Learning for Calibrated Decisions | TypeSafe AI’s training method for Jev; goal public (calibrated probabilities), mechanism undisclosed | 10 |
| RLCD (2023) | Reinforcement Learning from Contrastive Distillation | an unrelated earlier method: preference pairs from positive vs. negative prompts | 10 |
| RLCR | RL with Calibration Rewards | correctness plus Brier score on a verbalized confidence | 10 |
| RLHF | RL from Human Feedback | a pipeline: SFT, a reward model from human comparisons, RL (usually PPO) with a KL penalty | 6 |
| RLOO | REINFORCE Leave-One-Out | REINFORCE with the mean reward of the other samples as baseline | 2, 8 |
| RLVR | RL with Verifiable Rewards | RL against a rule-based checker; the term is from Tülu 3 | 5 |
| RM | Reward Model | a network trained (usually by Bradley–Terry) to score responses | 5 |
| RWR | Reward-Weighted Regression | supervised learning on samples weighted by (exponentiated) reward | 9 |
| SFT | Supervised Fine-Tuning | maximum likelihood on demonstrations; behavior cloning | 4 |
| SimPO | Simple Preference Optimization | reference-free, length-normalized DPO-like loss with a margin | 7 |
| STaR | Self-Taught Reasoner | keep self-generated rationales that reach the right answer and fine-tune on them, iterated | 9 |
| TD | Temporal Difference | bootstrapped value targets \(r + \gamma V(s')\) | 1 |
| TRPO | Trust Region Policy Optimization | maximize a surrogate subject to a KL constraint to the old policy | 3 |