Appendix E — Timeline

A dated lineage of the ideas in this book. Each year is the year of the primary source as recorded in the repository’s research-log.md (the arXiv submission year, unless a venue year is the usual citation). The bars on the right count the methods of the unified table by year.

Figure E.1: The ideas in the book in date order, colored by family, from REINFORCE (1992) to RLCD (2026), and the number of named variants per year in the unified table. A few classical ideas (the policy gradient, trust regions, behavior cloning, reward-weighted regression) were recombined over three decades; the burst of named variants after 2023 followed the KL-regularized target and verifiable rewards, which made new routes to it cheap to try.

Year	Milestone	Chapter
1992	REINFORCE (Williams 1992)	Policy Gradients
2007	Reward-weighted regression (Peters and Schaal 2007)	Imitating a Better Distribution
2011	DAgger (Ross et al. 2011)	The LLM as a Policy
2015	TRPO (Schulman, Levine, et al. 2015); GAE (Schulman, Moritz, et al. 2015)	Keeping Updates Stable, Policy Gradients
2016	Sequence-level KD (Kim and Rush 2016); REINFORCE for text generation (Ranzato et al. 2016)	Imitating, The LLM as a Policy
2017	PPO (Schulman et al. 2017); RL from human preferences (Christiano et al. 2017); KL-control (Jaques et al. 2017)	Stable Updates, Rewards, Shared Target
2019	RLHF on language models (Ziegler et al. 2019); AWR (Peng et al. 2019)	Shared Target
2020	Learning to summarize from human feedback (Stiennon et al. 2020)	Shared Target
2022	InstructGPT (Ouyang et al. 2022); Constitutional AI (Bai et al. 2022); STaR (Zelikman et al. 2022); RL as inference (Korbak et al. 2022); RM overoptimization (Gao et al. 2022)	several
2023	DPO (Rafailov et al. 2023); IPO (Azar et al. 2023); RAFT (Dong et al. 2023); ReST (Gulcehre et al. 2023); ReMax (Li et al. 2023); GKD (Agarwal et al. 2023); process supervision (Lightman et al. 2023)	Direct Preference, Imitating
2024	KTO (Ethayarajh et al. 2024); ORPO (Hong et al. 2024); SimPO (Meng et al. 2024); online DPO (Guo et al. 2024); RLOO for RLHF (Ahmadian et al. 2024); GRPO (Shao et al. 2024); RLVR (Lambert et al. 2024)	Direct Preference, Critic-free
2025	DeepSeek-R1 (DeepSeek-AI 2025); Dr. GRPO (Liu et al. 2025); DAPO (Yu et al. 2025); GSPO (Zheng et al. 2025); CISPO (MiniMax 2025); on-policy distillation (Lu and Thinking Machines Lab 2025); RLCR (Damani et al. 2025)	Critic-free, Imitating, Calibration
2026	RLCD (TypeSafe AI; mechanism undisclosed) (TypeSafe AI 2026)	Calibration

Agarwal, Rishabh, Nino Vieillard, Yongchao Zhou, et al. 2023. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. https://arxiv.org/abs/2306.13649.

Ahmadian, Arash, Chris Cremer, Matthias Gallé, et al. 2024. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. https://arxiv.org/abs/2402.14740.

Azar, Mohammad Gheshlaghi, Mark Rowland, Bilal Piot, et al. 2023. A General Theoretical Paradigm to Understand Learning from Human Preferences. https://arxiv.org/abs/2310.12036.

Bai, Yuntao, Saurav Kadavath, Sandipan Kundu, et al. 2022. Constitutional AI: Harmlessness from AI Feedback. https://arxiv.org/abs/2212.08073.

Christiano, Paul, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. https://arxiv.org/abs/1706.03741.

Damani, Mehul, Isha Puri, Stewart Slocum, et al. 2025. Beyond Binary Rewards: Training LMs to Reason about Their Uncertainty. https://arxiv.org/abs/2507.16806.

DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. https://arxiv.org/abs/2501.12948.

Dong, Hanze, Wei Xiong, Deepanshu Goyal, et al. 2023. RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment. https://arxiv.org/abs/2304.06767.

Ethayarajh, Kawin, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. KTO: Model Alignment as Prospect Theoretic Optimization. https://arxiv.org/abs/2402.01306.

Gao, Leo, John Schulman, and Jacob Hilton. 2022. Scaling Laws for Reward Model Overoptimization. https://arxiv.org/abs/2210.10760.

Gulcehre, Caglar, Tom Le Paine, Srivatsan Srinivasan, et al. 2023. Reinforced Self-Training (ReST) for Language Modeling. https://arxiv.org/abs/2308.08998.

Guo, Shangmin, Biao Zhang, Tianlin Liu, et al. 2024. Direct Language Model Alignment from Online AI Feedback. https://arxiv.org/abs/2402.04792.

Hong, Jiwoo, Noah Lee, and James Thorne. 2024. ORPO: Monolithic Preference Optimization without Reference Model. https://arxiv.org/abs/2403.07691.

Jaques, Natasha, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E. Turner, and Douglas Eck. 2017. “Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-Control.” Proceedings of the 34th International Conference on Machine Learning (ICML), 1645–54. https://arxiv.org/abs/1611.02796.

Kim, Yoon, and Alexander M. Rush. 2016. “Sequence-Level Knowledge Distillation.” Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). https://arxiv.org/abs/1606.07947.

Korbak, Tomasz, Ethan Perez, and Christopher L Buckley. 2022. RL with KL penalties is better viewed as Bayesian inference. https://arxiv.org/abs/2205.11275.

Lambert, Nathan, Jacob Morrison, Valentina Pyatkin, et al. 2024. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. https://arxiv.org/abs/2411.15124.

Li, Ziniu, Tian Xu, Yushun Zhang, et al. 2023. ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. https://arxiv.org/abs/2310.10505.

Lightman, Hunter, Vineet Kosaraju, Yura Burda, et al. 2023. Let’s Verify Step by Step. https://arxiv.org/abs/2305.20050.

Liu, Zichen, Changyu Chen, Wenjun Li, et al. 2025. Understanding R1-Zero-Like Training: A Critical Perspective. https://arxiv.org/abs/2503.20783.

Lu, Kevin, and Thinking Machines Lab. 2025. On-Policy Distillation. https://thinkingmachines.ai/blog/on-policy-distillation/.

Meng, Yu, Mengzhou Xia, and Danqi Chen. 2024. SimPO: Simple Preference Optimization with a Reference-Free Reward. https://arxiv.org/abs/2405.14734.

MiniMax. 2025. MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. https://arxiv.org/abs/2506.13585.

Ouyang, Long, Jeff Wu, Xu Jiang, et al. 2022. Training language models to follow instructions with human feedback. https://arxiv.org/abs/2203.02155.

Peng, Xue Bin, Aviral Kumar, Grace Zhang, and Sergey Levine. 2019. Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning. https://arxiv.org/abs/1910.00177.

Peters, Jan, and Stefan Schaal. 2007. “Reinforcement Learning by Reward-Weighted Regression for Operational Space Control.” Proceedings of the 24th International Conference on Machine Learning (ICML), 745–50. https://doi.org/10.1145/1273496.1273590.

Rafailov, Rafael, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. https://arxiv.org/abs/2305.18290.

Ranzato, Marc’Aurelio, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. “Sequence Level Training with Recurrent Neural Networks.” International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1511.06732.

Ross, Stéphane, Geoffrey J. Gordon, and J. Andrew Bagnell. 2011. “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning.” Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS). https://arxiv.org/abs/1011.0686.

Schulman, John, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. 2015. Trust Region Policy Optimization. https://arxiv.org/abs/1502.05477.

Schulman, John, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015. High-Dimensional Continuous Control Using Generalized Advantage Estimation. https://arxiv.org/abs/1506.02438.

Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. https://arxiv.org/abs/1707.06347.

Shao, Zhihong, Peiyi Wang, Qihao Zhu, et al. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. https://arxiv.org/abs/2402.03300.

Stiennon, Nisan, Long Ouyang, Jeff Wu, et al. 2020. Learning to summarize from human feedback. https://arxiv.org/abs/2009.01325.

TypeSafe AI. 2026. Introducing System One Models & Jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev.

Williams, Ronald J. 1992. “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning.” Machine Learning 8: 229–56. https://doi.org/10.1007/BF00992696.

Yu, Qiying, Zheng Zhang, Ruofei Zhu, et al. 2025. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. https://arxiv.org/abs/2503.14476.

Zelikman, Eric, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. STaR: Bootstrapping Reasoning With Reasoning. https://arxiv.org/abs/2203.14465.

Zheng, Chujie, Shixuan Liu, Mingze Li, et al. 2025. Group Sequence Policy Optimization. https://arxiv.org/abs/2507.18071.

Ziegler, Daniel M., Nisan Stiennon, Jeffrey Wu, et al. 2019. Fine-Tuning Language Models from Human Preferences. https://arxiv.org/abs/1909.08593.