Auxiliary loss

As explained in Math and Architectures of Deep Learning

An additional loss term computed from an intermediate network output to help optimize a deep model.

An auxiliary loss is an additional training signal computed from an intermediate network output. In GoogLeNet, intermediate Inception modules feed auxiliary classifiers: softmax produces predictions, and each classifier’s loss is computed against the ground truth. The final objective is a weighted sum of the auxiliary loss and the real loss, helping prevent the middle of a deep network from dying under vanishing gradients. section 11.2.2

The technique connects three architectural and optimization choices:

• Attach two auxiliary classifiers to intermediate Inception modules. section 11.2.2
• Compare each intermediate prediction with the ground truth after applying softmax. section 11.2.2
• Combine auxiliary and real losses through a weighted sum. section 11.2.2

Figure 11.8 Inception_v1 architecture

Use the Inception_v1 architecture diagram to locate intermediate classifier outputs and distinguish them from the final prediction path. section 11.2.2

How Math and Architectures of Deep Learning covers it

How deep it goes

Defines intermediate classifiers and weighted loss aggregation, but defers the original paper’s implementation details.

Examples

Uses GoogLeNet with two auxiliary classifiers attached to intermediate Inception modules.

What you can do after

Understand how intermediate predictions contribute additional supervision to a deep image-classification objective.

How it connects

Connects Inception modules, auxiliary classifiers, softmax, ground truth, weighted sums, and vanishing gradients.

liveBookmanning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Related concepts

• objective functionpart of · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
• coefficient of variationuses · The coefficient of variation is a normalized measure of relative variability equal to a quantity’s standard deviation divided by its mean.
• guided attentionused by · Guided attention is a training technique that adds a penalty encouraging an attention matrix to follow a desired alignment pattern.
• hyperparameterconfigured by · A hyperparameter is a setting chosen outside ordinary parameter learning that controls a model, algorithm, or learning procedure.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.