Appendix D — Further Reading

Short and annotated. The chapters cite everything they use; these are the sources most worth reading in full.

D.1 Textbooks

  • Jorge Nocedal and Stephen Wright, Numerical Optimization (Nocedal and Wright 2006): the standard reference for gradient, Newton, quasi-Newton, conjugate-gradient and constrained methods, with proofs and practical advice. The source for most of
  • Stephen Boyd and Lieven Vandenberghe, Convex Optimization (Boyd and Vandenberghe 2004): convexity, duality and interior-point methods, with many modeling examples. Free online. The source for most of Chapter 5.
  • Steven Brunton, Optimization Bootcamp (Brunton 2026): a lecture series covering much of this book’s ground with an applied, visual emphasis; it was the starting point for this project.
  • Christopher Bishop, Pattern Recognition and Machine Learning (Bishop 2006): maximum likelihood, the Laplace approximation and EM, as used in Chapter 7.
  • Carl Rasmussen and Christopher Williams, Gaussian Processes for Machine Learning (Rasmussen and Williams 2006): the surrogate model behind Bayesian optimization.
  • George Dantzig, Linear Programming and Extensions (Dantzig 1963): the origin of the simplex method, by its inventor.

D.2 Papers worth reading in full

  • Stephen Boyd and coauthors on ADMM (Boyd et al. 2011): a long, readable survey of splitting methods and their applications.
  • Amir Beck and Marc Teboulle on FISTA (Beck and Teboulle 2009): proximal gradient and its accelerated version, with clear proofs.
  • Donald Jones, Matthias Schonlau and William Welch on efficient global optimization (Jones et al. 1998): the paper that made Bayesian optimization with expected improvement standard.
  • Yann Dauphin and coauthors on saddle points (Dauphin et al. 2014), and Hao Li and coauthors on visualizing loss landscapes (Li et al. 2018): two entry points to the questions of Chapter 8.
  • Arthur Jacot, Franck Gabriel and Clément Hongler on the neural tangent kernel (Jacot et al. 2018): why very wide networks train like linear models.
Beck, Amir, and Marc Teboulle. 2009. “A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems.” SIAM Journal on Imaging Sciences 2 (1): 183–202. https://doi.org/10.1137/080716542.
Bishop, Christopher M. 2006. Pattern Recognition and Machine Learning. Springer.
Boyd, Stephen, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. 2011. “Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers.” Foundations and Trends in Machine Learning 3 (1). https://web.stanford.edu/~boyd/papers/admm_distr_stats.html.
Boyd, Stephen, and Lieven Vandenberghe. 2004. Convex Optimization. Cambridge University Press. https://web.stanford.edu/~boyd/cvxbook/.
Brunton, Steven L. 2026. Optimization: A Bootcamp for Machine Learning, Inverse Problems, and Control. Cambridge University Press.
Dantzig, George B. 1963. Linear Programming and Extensions. Princeton University Press.
Dauphin, Yann N., Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. 2014. “Identifying and Attacking the Saddle Point Problem in High-Dimensional Non-Convex Optimization.” Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/1406.2572.
Jacot, Arthur, Franck Gabriel, and Clément Hongler. 2018. “Neural Tangent Kernel: Convergence and Generalization in Neural Networks.” Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/1806.07572.
Jones, Donald R., Matthias Schonlau, and William J. Welch. 1998. “Efficient Global Optimization of Expensive Black-Box Functions.” Journal of Global Optimization 13 (4): 455–92. https://doi.org/10.1023/A:1008306431147.
Li, Hao, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018. “Visualizing the Loss Landscape of Neural Nets.” Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/1712.09913.
Nocedal, Jorge, and Stephen J. Wright. 2006. Numerical Optimization. 2nd ed. Springer.
Rasmussen, Carl Edward, and Christopher K. I. Williams. 2006. Gaussian Processes for Machine Learning. MIT Press.