liveBook · Manning

Auxiliary loss

An auxiliary loss is an additional training objective or loss term optimized alongside a primary objective to encourage a desired property or improve learning.

How 2 Manning books explain it

Build a DeepSeek Model (From Scratch)

An auxiliary loss is an additional penalty term optimized alongside a model's main training loss to encourage a desired property such as balanced expert selection.

How it is used
Encourages balanced expert importance by adding a routing penalty to the main next-token prediction objective.
How deep it goes
Defines column-summed importance, derives coefficient of variation, and gives a squared penalty, but stops short of implementation code.
Examples
Uses three expert importance scores of 1.4, 1.0, and 1.6, plus a two-expert case exposing unequal token loads.
What you can do after
Apply it via a coefficient-of-variation penalty added to a mixture-of-experts training loss.

Read how Build a DeepSeek Model (From Scratch) explains auxiliary loss →

Math and Architectures of Deep Learning

An additional loss term computed from an intermediate network output to help optimize a deep model.

How it is used
Uses intermediate supervision to keep very deep image-classification networks training effectively.
How deep it goes
Defines intermediate classifiers and weighted loss aggregation, but defers the original paper’s implementation details.
Examples
Uses GoogLeNet with two auxiliary classifiers attached to intermediate Inception modules.
What you can do after
Understand how intermediate predictions contribute additional supervision to a deep image-classification objective.

Read how Math and Architectures of Deep Learning explains auxiliary loss →

What to know first

Related concepts

Questions about auxiliary loss

What is auxiliary loss?

An auxiliary loss is an additional training objective or loss term optimized alongside a primary objective to encourage a desired property or improve learning.

Which Manning books explain auxiliary loss?

Build a DeepSeek Model (From Scratch) and Math and Architectures of Deep Learning.

What should I know before learning auxiliary loss?

Manning books that teach auxiliary loss usually introduce objective function and top-k first.

What is the difference between auxiliary loss and load-balancing loss?

An auxiliary loss is an additional training objective or loss term optimized alongside a primary objective to encourage a desired property or improve learning. A load-balancing loss is an auxiliary objective that encourages routing assignments to be distributed evenly across mixture-of-experts modules.

What is auxiliary loss part of?

Auxiliary loss is part of objective function. An objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.