Auxiliary loss
An auxiliary loss is an additional training objective or loss term optimized alongside a primary objective to encourage a desired property or improve learning.
Build a DeepSeek Model (From Scratch)
An auxiliary loss is an additional penalty term optimized alongside a model's main training loss to encourage a desired property such as balanced expert selection.
- How it is used
- Encourages balanced expert importance by adding a routing penalty to the main next-token prediction objective.
- How deep it goes
- Defines column-summed importance, derives coefficient of variation, and gives a squared penalty, but stops short of implementation code.
- Examples
- Uses three expert importance scores of 1.4, 1.0, and 1.6, plus a two-expert case exposing unequal token loads.
- What you can do after
- Apply it via a coefficient-of-variation penalty added to a mixture-of-experts training loss.
Read how Build a DeepSeek Model (From Scratch) explains auxiliary loss →
Math and Architectures of Deep Learning
An additional loss term computed from an intermediate network output to help optimize a deep model.
- How it is used
- Uses intermediate supervision to keep very deep image-classification networks training effectively.
- How deep it goes
- Defines intermediate classifiers and weighted loss aggregation, but defers the original paper’s implementation details.
- Examples
- Uses GoogLeNet with two auxiliary classifiers attached to intermediate Inception modules.
- What you can do after
- Understand how intermediate predictions contribute additional supervision to a deep image-classification objective.
Read how Math and Architectures of Deep Learning explains auxiliary loss →
What to know first
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- top-kTop-k is a selection operation that retains the k items ranked highest according to a specified value or criterion.
Related concepts
- objective functionpart of · An objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- coefficient of variationuses · The coefficient of variation is a normalized measure of relative variability equal to a quantity’s standard deviation divided by its mean.
- guided attentionused by · Guided attention is a training technique that adds a penalty encouraging an attention matrix to follow a desired alignment pattern.
- load-balancing losscontrasts with · A load-balancing loss is an auxiliary objective that encourages routing assignments to be distributed evenly across mixture-of-experts modules.
- hyperparameterconfigured by · A hyperparameter is a setting chosen outside ordinary parameter learning that controls a model, algorithm, or learning procedure.
Questions about auxiliary loss
What is auxiliary loss?
An auxiliary loss is an additional training objective or loss term optimized alongside a primary objective to encourage a desired property or improve learning.
Which Manning books explain auxiliary loss?
Build a DeepSeek Model (From Scratch) and Math and Architectures of Deep Learning.
What should I know before learning auxiliary loss?
Manning books that teach auxiliary loss usually introduce objective function and top-k first.
What is the difference between auxiliary loss and load-balancing loss?
An auxiliary loss is an additional training objective or loss term optimized alongside a primary objective to encourage a desired property or improve learning. A load-balancing loss is an auxiliary objective that encourages routing assignments to be distributed evenly across mixture-of-experts modules.
What is auxiliary loss part of?
Auxiliary loss is part of objective function. An objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

