9 Augmentation
Augmentation creates training data from training data: flip an image, crop it, mask part of a spectrogram, and train on the copies as if they were new. It is usually presented as a bag of tricks that “add data”. Each trick carries an assumption, though: that the transformed copy has a label you know. That assumption is a claim about the target distribution \(p\), and it can be false. This chapter treats augmentation as what it is on the four levers, a change to the sampling distribution \(q\) that is justified only by an assumption, and measures what happens when the assumption is true, false, and false but known.
9.1 The problem
Given a transform \(T\) of the inputs (a flip, a shift, a mask), training on the pairs \((T(x), y)\) adds them to \(q\). Does that help, and when can it hurt? And what about transforms that change the label, or mix two examples together?
9.2 Assumptions
- A label-preserving augmentation assumes invariance: \(p(y \mid T(x)) = p(y \mid x)\) for every transform used, at every strength used. If that fails, the augmented pairs are mislabeled examples.
- A label-changing augmentation assumes a known effect: the label of \(T(x)\) is a known function of the label of \(x\).
- The demonstration uses a label with exactly known symmetries. As Data Types and Their Structure noted, real labels rarely have invariances that hold exactly. T5’s synthetic “side” label does: on the bundled digit images, it is 1 when the left half holds more ink than the right. An up–down flip leaves it unchanged; a left–right flip reverses it (The Testbeds).
9.3 Derivation or demonstration
Augmentation is a change to \(q\). Training on the original pairs plus their transforms changes the training distribution to a mixture of \(q\) and its image under \(T\). If assumption 1 holds, every added pair is a pair that \(p\) could have produced. The model then sees more of the target distribution, and its estimate varies less. If assumption 1 fails, the added pairs carry labels \(p\) would not give, which is label noise of a systematic kind (Labels).
Demonstration on T5. A logistic regression (scikit-learn’s defaults, \(C = 1\)) is trained on \(n\) of the 1,783 digit images with a tie-free side label, and tested on the rest; 20 seeds. Four training sets:
- no augmentation;
- up–down flips with the same label: a true invariance;
- left–right flips with the same label: a false invariance, since the flip reverses the label;
- left–right flips with the label reversed: a label-changing augmentation whose effect is known.
With \(n = 10\) images, test accuracy was 0.73 without augmentation, 0.76 with the true invariance, 0.85 with the label-changing flip, and 0.57 with the false invariance. With \(n = 400\) the four were 0.961, 0.977, 0.979 and 0.67 (tests/test_augmentation.py::test_invariant_augmentation_helps_noninvariant_hurts and ::test_label_changing_augmentation_helps_when_the_change_is_known).
Augmenting with transforms the target is invariant to adds examples \(p\) could have produced. It never hurt here, and helped most when data was scarce. Augmenting with a transform the target is not invariant to, keeping the label, adds systematically wrong labels; here it hurt at every training size, and more data did not close the gap. A transform that changes the label is valuable once the change is known: labeling the copies accordingly turned the most damaging augmentation into the most useful one.
| Changes | Sampling distribution \(q\): adds transformed copies of training inputs with their labels unchanged |
| Assumption | The target is invariant under the transform: \(p(y \mid T(x)) = p(y \mid x)\) for every transform \(T\) used, at the strengths used |
| What it does to the learned function | More training data consistent with \(p\), so less variance; the model is pushed toward the assumed invariance |
| Which models care | Every model; most valuable at small \(n\) and for models that cannot build the invariance in |
| Fit on | Training data only, transformed during training |
| Failure modes | The invariance is false, or false beyond some strength: augmented examples are then mislabeled; test-time inputs never look like the augmented ones |
| Alternatives | An architecture that has the invariance built in; label-changing augmentation when the transform’s effect is known |
| Checked by | tests/test_augmentation.py::test_invariant_augmentation_helps_noninvariant_hurts |
| Changes | Sampling distribution \(q\) and labels: adds transformed inputs with labels changed as the transform is known to change them |
| Assumption | The transform’s effect on the label is known exactly (here: a left-right flip reverses which half holds more ink) |
| What it does to the learned function | As much new, correctly labeled data as a label-preserving augmentation, including examples near the boundary from the other side |
| Which models care | Every model |
| Fit on | Training data only |
| Failure modes | A wrongly assumed label change is label noise by construction; the effect may be known only for some inputs |
| Alternatives | Label-preserving augmentation with a different transform; collecting the transformed cases |
| Checked by | tests/test_augmentation.py::test_label_changing_augmentation_helps_when_the_change_is_known |
Augmentation policies. In practice, augmentations are applied as randomized policies. RandAugment reduces a learned policy to two numbers: “N: Number of augmentation transformations to apply sequentially” and “M: Magnitude for all the transformations”, applied to transforms drawn from a fixed list of 14 (Cubuk et al. 2019, fig. 2). SpecAugment acts on the log mel spectrogram of speech with three transforms. Time warping deforms the time axis. Frequency masking blanks \(f\) consecutive mel channels, with \(f\) drawn uniformly from 0 to a parameter \(F\). Time masking blanks \(t\) consecutive time steps, with \(t\) drawn uniformly from 0 to a parameter \(T\) (Park et al. 2019, sec. 2). Both control the strength of the invariance assumption with a single magnitude, and the result above says why the strength matters. A transform the target tolerates when mild can change it when strong: a small shift keeps a digit, while a large one can push it out of the frame.
mixup. mixup trains on mixtures of pairs: \(\tilde x = \lambda x_i + (1 - \lambda)\,x_j\) and \(\tilde y = \lambda y_i + (1 - \lambda)\,y_j\), with \(\lambda \sim \mathrm{Beta}(\alpha, \alpha)\). Its authors frame it as a “vicinal” distribution around the training data, in the Vicinal Risk Minimization tradition (Zhang et al. 2017, sec. 2). On the levers it changes both \(q\) (new inputs between examples) and the labels (soft ones). Its assumption is different from a symmetry: the target changes linearly along the segment between two examples. A test checks the implementation against that definition (tests/test_augmentation.py::test_mixup_forms_the_same_convex_combination_of_inputs_and_labels).
| Changes | Sampling distribution \(q\) and labels: trains on convex combinations of pairs of inputs, with the same combination of their labels |
| Assumption | Between two training examples, the target changes linearly along the segment joining them (a vicinal distribution around the data) |
| What it does to the learned function | Trains the model on targets that change linearly between examples, so the learned function is pushed toward that behavior between them |
| Which models care | Models trained by gradient on a loss that accepts soft labels, mostly neural networks |
| Fit on | Training batches, with the mixing weight drawn from Beta(\(\alpha\), \(\alpha\)) |
| Failure modes | Mixed inputs that are not plausible inputs; targets that are not linear between examples; small \(\alpha\) changes little, large \(\alpha\) blurs classes |
| Alternatives | Label-preserving augmentation; label smoothing |
| Checked by | tests/test_augmentation.py::test_mixup_forms_the_same_convex_combination_of_inputs_and_labels |
Apply augmentation to training batches only, inside each fold, never to validation or test data. Choose transforms by asking, for each one and each strength, whether the label provably survives. Where a transform changes the label in a known way, apply the change instead of dropping the transform.
9.4 What it does to the learned model
A label-preserving augmentation that is true is the cheapest data there is. It moves the model toward the invariance it encodes and reduces variance, which matters most when \(n\) is small. A false one teaches the model a contradiction: in the demonstration, every image and its mirror carry the same label although they have opposite true labels. The model can no longer separate the two cases. Its accuracy then stayed between 0.56 and 0.67 across training sizes, against 0.5 for guessing.
9.5 Failure modes
Figure 9.1 shows the four training sets side by side. The failures that generalize:
- A symmetry the task does not have. A horizontal flip that suits many photographs is wrong for text, digits and anything with handedness.
- A symmetry that holds only at small strength. Rotating a 6 slightly keeps it a 6; rotating it by half a turn makes it a 9. Policies with a single magnitude parameter make this a tuning choice, but the choice is still an assumption about \(p\).
- Augmenting the evaluation data. Test-time inputs are draws from \(p\); transforming them measures something else.
9.6 What the toy cannot show
The side label is exactly symmetric, so whether an augmentation is right can be decided by construction. Real symmetries are approximate, partial and strength-dependent, and nothing measures them except held-out data. The demonstration uses a linear model on tiny images. Networks that learn their own features may gain more, or less, from the same augmentation.
9.7 Connections
- Data Types and Their Structure: the plausible invariances of each data type, which augmentation turns into training data.
- Labels: a false invariance creates label noise that depends on \(x\), the hardest kind to detect.
- Sampling and Weighting: augmentation and SMOTE both add synthetic examples, under different assumptions about where the label is known.
- loss-functions-lab, Contrastive Objectives: augmentation as the source of positive pairs in self-supervised learning.