Appendix F — Further Reading

Short and annotated. The chapters cite everything they use; these are the sources most worth reading in full.

F.1 Classical RL

• Richard Sutton and Andrew Barto, Reinforcement Learning: An Introduction, 2nd ed. (Sutton and Barto 2018). Chapter 2 (bandits, evaluative feedback), chapter 13 (policy gradients) and section 11.3 (the deadly triad) are the background this book assumes.
• John Schulman et al. on TRPO, GAE and PPO (Schulman, Levine, et al. 2015; Schulman, Moritz, et al. 2015; Schulman et al. 2017): three short papers that define the policy-gradient machinery LLM training still uses. Schulman’s note on approximating KL divergence (Schulman 2020) is two pages and explains \(k_1\), \(k_2\) and \(k_3\).
• Sergey Levine, “Reinforcement Learning and Control as Probabilistic Inference” (Levine 2018), for the inference view behind the closed-form optimum.

F.2 The RLHF lineage

• Christiano et al. (Christiano et al. 2017), Ziegler et al. (Ziegler et al. 2019), Stiennon et al. (Stiennon et al. 2020) and Ouyang et al. (Ouyang et al. 2022), in that order: the recipe as it was built.
• Korbak, Perez and Buckley (Korbak et al. 2022): why the KL penalty makes the objective a posterior.
• Gao, Schulman and Hilton (Gao et al. 2022): overoptimization measured.

F.3 Beyond PPO

• Rafailov et al. (Rafailov et al. 2023) for DPO, and Azar et al. (Azar et al. 2023) for the family it belongs to and its overfitting failure mode.
• Ahmadian et al. (Ahmadian et al. 2024): the case for simple REINFORCE-style estimators in RLHF.
• Shao et al. (Shao et al. 2024), section 5.2: the unified view of SFT, rejection sampling, DPO, PPO and GRPO by data source, reward and gradient coefficient, which this book’s \((q, w)\) form follows.
• Liu et al. (Z. Liu et al. 2025) and Yu et al. (Yu et al. 2025): what GRPO’s choices do, and what to change.

F.4 What RL does to a model

• Yue et al. (Yue et al. 2025) and Liu et al. (M. Liu et al. 2025) read together, as the two sides of the pass@\(k\) debate.
• Shenfeld, Pari and Agrawal (Shenfeld et al. 2025) on forgetting, and Kirk et al. (Kirk et al. 2023) on generalization and diversity.

F.5 This author’s other books

• loss-functions-lab: where Bradley–Terry, the DPO derivation and proper scoring rules are derived as losses.
• modern-ai-systems-and-methods: the short overview of RL, bandits and generative AI that this book expands.
• transformer-atlas: the architectures, which this book treats as given.

Ahmadian, Arash, Chris Cremer, Matthias Gallé, et al. 2024. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. https://arxiv.org/abs/2402.14740.

Azar, Mohammad Gheshlaghi, Mark Rowland, Bilal Piot, et al. 2023. A General Theoretical Paradigm to Understand Learning from Human Preferences. https://arxiv.org/abs/2310.12036.

Christiano, Paul, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. https://arxiv.org/abs/1706.03741.

Gao, Leo, John Schulman, and Jacob Hilton. 2022. Scaling Laws for Reward Model Overoptimization. https://arxiv.org/abs/2210.10760.

Kirk, Robert, Ishita Mediratta, Christoforos Nalmpantis, et al. 2023. Understanding the Effects of RLHF on LLM Generalisation and Diversity. https://arxiv.org/abs/2310.06452.

Korbak, Tomasz, Ethan Perez, and Christopher L Buckley. 2022. RL with KL penalties is better viewed as Bayesian inference. https://arxiv.org/abs/2205.11275.

Levine, Sergey. 2018. Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review. https://arxiv.org/abs/1805.00909.

Liu, Mingjie, Shizhe Diao, Ximing Lu, et al. 2025. ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. https://arxiv.org/abs/2505.24864.

Liu, Zichen, Changyu Chen, Wenjun Li, et al. 2025. Understanding R1-Zero-Like Training: A Critical Perspective. https://arxiv.org/abs/2503.20783.

Ouyang, Long, Jeff Wu, Xu Jiang, et al. 2022. Training language models to follow instructions with human feedback. https://arxiv.org/abs/2203.02155.

Rafailov, Rafael, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. https://arxiv.org/abs/2305.18290.

Schulman, John. 2020. Approximating KL Divergence. http://joschu.net/blog/kl-approx.html.

Schulman, John, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. 2015. Trust Region Policy Optimization. https://arxiv.org/abs/1502.05477.

Schulman, John, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015. High-Dimensional Continuous Control Using Generalized Advantage Estimation. https://arxiv.org/abs/1506.02438.

Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. https://arxiv.org/abs/1707.06347.

Shao, Zhihong, Peiyi Wang, Qihao Zhu, et al. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. https://arxiv.org/abs/2402.03300.

Shenfeld, Idan, Jyothish Pari, and Pulkit Agrawal. 2025. RL’s Razor: Why Online Reinforcement Learning Forgets Less. https://arxiv.org/abs/2509.04259.

Stiennon, Nisan, Long Ouyang, Jeff Wu, et al. 2020. Learning to summarize from human feedback. https://arxiv.org/abs/2009.01325.

Sutton, Richard S., and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. 2nd ed. MIT Press. http://incompleteideas.net/book/the-book-2nd.html.

Yu, Qiying, Zheng Zhang, Ruofei Zhu, et al. 2025. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. https://arxiv.org/abs/2503.14476.

Yue, Yang, Zhiqi Chen, Rui Lu, et al. 2025. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? https://arxiv.org/abs/2504.13837.

Ziegler, Daniel M., Nisan Stiennon, Jeffrey Wu, et al. 2019. Fine-Tuning Language Models from Human Preferences. https://arxiv.org/abs/1909.08593.