MixUp

As explained in Inside Deep Learning: Math, Algorithms, Models

A data augmentation and regularization technique that creates synthetic training examples by linearly mixing two inputs and their labels.

MixUp reduces overfitting by changing both the training inputs and loss instead of penalizing network weights. It blends two inputs with a random λ between 0 and 1, encouraging proportional predictions rather than overconfident class scores. section 14.3 section 14.5

Implement MixUp by treating a shuffled batch as its paired batch:

• Sample λ from Beta(α, α), where α controls mixing strength. section 14.3.2
• Shuffle a batch with torch.randperm(B), then compute x̃ = λx + (1 − λ)x[permutation]. section 14.3.2
• Keep both label tensors and calculate λℓ(ŷ, yi) + (1 − λ)ℓ(ŷ, yj). section 14.3.2
• Place mixing in a DataLoader collate function and wrap the ordinary loss; the optimizer and scheduler can remain unchanged. section 14.3.2

PyTorch training setup

Use DataLoader(..., collate_fn=MixupCollator()), then train with MixupLoss(loss). A collator samples λ, applies torch.randperm, mixes inputs, and returns the original and shuffled labels. section 14.3.2

Figure 14.7 With MixUp, there is not the same incentive to overpredict. Instead, the model tries to reach a target percentage based on how much of two different classes was selected. MixUp mixes the labels and the inputs so there is a signal for each class proportional to the MixUp percentage λ.

Shows how MixUp changes the target from an overconfident single-class prediction to percentages proportional to λ. section 14.3

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes

Defines the mixing equations and implements them through a loss wrapper, batch permutation, and DataLoader collator.

Examples

Uses shuffled image batches, mixed labels, MixupLoss, MixupCollator, and accuracy comparisons against regular and residual models.

What it assumes

Assumes ordinary loss-based neural-network training and introduces paired labels, weighted losses, permutations, and batch collation.

What you can do after

Apply MixUp via PyTorch by configuring a collator, wrapping the base loss, and training with the existing optimizer pipeline.

How it connects

Contrasts with dropout and connects to cross-entropy, regularization, DataLoader collators, and neural-network training.

liveBookmanning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Related concepts

• overfittingaddresses · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
• objective functionuses · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
• confidenceaddresses · Confidence is a degree of certainty or assurance associated with a judgment, estimate, prediction, capability, or statistical procedure, often expressed as a score, probability, or coverage level.
• collate functionuses · A collate function combines individual data samples into a batch, optionally transforming or padding them into a common format.
• DataLoaderuses · PyTorch DataLoader is an iterable utility that retrieves dataset samples, optionally shuffles and transforms them, and groups them into batches.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.