Appendix C — The Code Behind the Figures

Every figure and number in this book is computed when the book is built, by a small Python package, optimlab, that implements each method from scratch with NumPy and JAX. This appendix explains how that code is organized, how it is checked, and how to run it. Nothing in the chapters depends on reading it.

C.1 One interface for every method

Almost every method takes the same two things in and gives the same thing back. A problem bundles an objective \(f\), a starting point and, optionally, a search box and known minima; gradients and Hessians are computed by automatic differentiation (JAX) unless supplied by hand. A result reports the final point and value, the number of iterations, whether the method’s own stopping test was met, and the full history of iterates, which is what the figures draw. Because of this shared shape, any general-purpose method can run on any problem, which is what Choosing an Algorithm exploits. Methods for special problem classes (linear programs, least squares, composite and constrained problems) take their own problem types.

part of the package	contents	used in
core	the problem and result types, automatic differentiation	all
landscapes	test functions (sphere, Rosenbrock, Rastrigin, Himmelblau, …)	Chapter 1, Chapter 6
optimizers	every iterative method in the book	Chapter 2 to Chapter 6
linalg	SVD, least squares, ridge, equality-constrained QP	Chapter 1, Chapter 4, Chapter 5
inference	MLE, MAP, Laplace, Metropolis–Hastings, EM	Chapter 7
highdim	networks as problems, Lanczos, loss slices, mode connectivity, NTK	Chapter 8
inverse, control, ml, problems	the applications	Chapter 9
viz	every figure builder, sharing one colorblind-checked theme	all
backends	adapters to SciPy, CVXPY and Optuna, used as independent checks	Chapter 3 to Chapter 6

C.2 How the code is checked

Each method has its own test file, and the tests check real behavior against known answers rather than only that the code runs: convergence to known minima, agreement with closed-form solutions, and agreement with independent solvers (SciPy’s HiGHS for linear programs, CVXPY for constrained problems, Optuna for black-box search). The Checked by link at the end of each method section in the book points to its tests.

C.3 Running it yourself

The source is on GitHub. With uv installed:

uv sync --extra viz --extra dev --extra docs   # install
uv run pytest                                  # run the tests
quarto render docs                             # build this book (needs Quarto)

Interactive notebooks

Eight marimo notebooks let you change a problem or a setting and watch the figures update, one per topic:

notebook	explores	chapter
gradient_descent_explorer.py	step sizes, momentum and solver races on test landscapes	Chapter 2
lp_polytope_explorer.py	the simplex path on an editable linear program	Chapter 3
regularization_path_explorer.py	ridge and lasso paths	Chapter 4
central_path_explorer.py	the barrier method’s central path	Chapter 5
bayesian_posterior_explorer.py	Laplace versus MCMC on a coin-flip posterior	Chapter 7
loss_landscape_explorer.py	loss slices and Hessian spectra of small networks	Chapter 8
control_explorer.py	LQR cost weights and initial state, re-solved live	Chapter 9
solver_arena_explorer.py	every method on a chosen test function	Appendix A

uv run marimo edit notebooks/marimo/gradient_descent_explorer.py

The notebooks need a local Python: the package uses JAX, which cannot run inside a web browser, so they cannot be embedded in these pages.