25  Machine Learning

The map opened with an ML practitioner writing embedding_matrix @ x and wanting to know what mathematical world that equation actually lives in. This chapter is the answer — not a new theory, but an index: every piece of machinery a working ML practitioner touches traced back to the structure or theory that already defines it elsewhere on this site. Nothing here is taught for the first time.

25.1 Data is vectors, and “similar” is an angle

A learned embedding is an element of a vector space, in exactly Vector Spaces’s sense — not a metaphor, a literal instance. “Similarity” between two embeddings is almost always computed as cosine similarity, \(\frac{\langle u,v\rangle}{\|u\|\|v\|}\) — Inner Product Spaces’s angle, applied to word vectors, document vectors, or image embeddings instead of geometric arrows. Nothing about the formula changes; only what’s plugged into it does, which is the entire lesson Vector Spaces opened this site’s structural arc with.

25.2 A model is a composition of maps

\(Wx + b\) — a weight matrix times an input, plus a bias — is an affine map in precisely Affine Spaces’s sense: a linear map (\(Wx\), Linear Algebra’s subject) plus a translation (\(+b\)). A neural network layer composes one of these with a fixed nonlinearity (an activation function); a full network is a long composition of such layers, exactly the kind of function composition Linear Algebra built matrix multiplication to represent, with a nonlinearity inserted between each composition to keep the whole stack from collapsing into a single linear map (a composition of purely linear maps is always just another linear map — the nonlinearities are what let depth add expressive power at all). PCA, mentioned but not derived in Linear Algebra, is mechanically nothing more than finding a data set’s dominant eigenvectors — a direct application of that chapter’s eigenvalue machinery, not a separate technique.

25.3 Training is optimization, and backpropagation is the chain rule

Fitting a model means minimizing a loss function over its parameters — Optimization’s subject, directly. The gradient that gradient descent follows is defined exactly as Inner Product Spaces and Optimization already derived it. Backpropagation, the algorithm that computes that gradient efficiently through a deep network, is not a distinct piece of mathematics — it’s Differentiation’s chain rule, applied systematically to a long composition of maps, computed back to front so each layer’s contribution is reused rather than recomputed. The “derivative of a composition is a composition of derivatives” fact Differentiation built on linear maps is the entire algorithm; a computational-graph diagram is a bookkeeping device for applying it in the right order, not new mathematics.

25.4 Outputs are distributions, and loss is often a likelihood

A classifier’s output is standardly interpreted as a probability distribution over classes, making its prediction a random variable in the technical sense — a function from possible model states to outcomes, not a single guaranteed answer. Training by minimizing cross-entropy loss is, mechanically, maximizing the likelihood of the observed labels under the model — the same “fit by maximizing probability of the data” idea Probability closed with, wearing a different name.

25.5 Information theory measures distributions, not just samples from them

Entropy, \(H(X) = -\mathbb{E}[\log P(X)]\), is Probability’s expectation applied to \(-\log P(X)\) — the average “surprise” a distribution’s own outcomes carry. KL divergence, \(D_{KL}(P\|Q) = \mathbb{E}_P[\log \frac{P}{Q}]\), is the same construction measuring how different two distributions are, and it’s not a separate tool from the cross-entropy loss just introduced: cross-entropy decomposes as \(H(P,Q) = H(P) + D_{KL}(P\|Q)\), and since \(H(P)\) (the true label distribution’s own entropy) doesn’t depend on the model at all, minimizing cross-entropy loss is minimizing \(D_{KL}\) between the true and predicted distributions — the same likelihood-maximizing fit from the previous section, in different notation, not a third idea. Mutual information, \(I(X;Y) = D_{KL}\big(P(X,Y) \,\|\, P(X)P(Y)\big)\), measures how far two random variables are from Probability’s independence — zero exactly when they’re independent, and a natural generalization of correlation to relationships correlation can’t see (nonlinear ones, in particular).

25.6 Why high dimensions feel unintuitive

Data in ML typically lives in spaces with hundreds or thousands of dimensions, and geometric intuition built in \(\mathbb{R}^2\) or \(\mathbb{R}^3\) stops transferring reliably: volume concentrates near a high-dimensional sphere’s surface rather than its center, and most pairs of random points end up roughly equidistant. None of this contradicts Euclidean Geometry — it’s the same inner- product-space structure, just in a regime where low-dimensional intuition was never a proof of anything to begin with. The manifold hypothesis — that real high-dimensional data actually concentrates near a much lower- dimensional surface embedded in that space — is exactly Euclidean Geometry’s one-paragraph manifold sketch, now doing real explanatory work: dimensionality-reduction techniques are, at bottom, attempts to find that lower-dimensional surface.

25.7 Where structure and relationships matter

Data with an explicit relational structure — social networks, molecules, citation graphs — is modeled with Discrete Mathematics‘s graphs directly, with a learned function applied at each vertex that depends on its neighbors. This isn’t a special case bolted onto the rest of ML; it’s the same map-composition idea from earlier in this chapter, with the graph’s edge structure deciding which vertices’ values get combined at each step instead of a fixed grid or sequence.

25.8 If you’ve trained a model

You already know the loop: forward pass, compute the loss, backward pass, step the optimizer. What this chapter adds, all in one place:

• the forward pass is a composition of affine maps (Affine Spaces) and fixed nonlinearities;
• the loss is, in an enormous share of practical cases, a cross-entropy — which is a KL divergence between the true and predicted distributions in different notation, not a separate quantity (Probability);
• the backward pass is the chain rule, computed back-to-front (Differentiation);
• the optimizer step is one discretized step of a gradient flow (Differential Equations, Optimization).

None of this changes how the code is written. It changes what the code is doing.

25.9 Where this leads

This closes the loop the map opened this entire site with. What’s left, deliberately kept light rather than developed in full — Advanced Topics, next — is the machinery behind the parts of ML this chapter only gestured at: infinite-dimensional function spaces behind kernel methods and Gaussian processes, the measure theory behind fully rigorous continuous probability, the differential geometry behind manifold learning done properly, and the increasingly common categorical language for describing how all of it composes.