liveBook · Manning

Attention subnetwork

As explained in Inside Deep Learning: Math, Algorithms, Models

A smaller subnetwork that produces a score for each input item and converts those scores into attention weights.

An attention subnetwork scores each extracted item representation and converts the scores into normalized weights. The weights determine each item’s contribution to a weighted average. section 10.1.3

A practical implementation separates three operations:

  • A backbone or feature-extraction network maps each input item to a representation hᵢ. section 10.1.3

Weighting digits in a bag

For a bag containing three digit images, the attention subnetwork assigns one normalized weight to each digit. Plotting the images with their weights makes the selected evidence visible before classification.

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes
Defines the subnetwork architecture and tensor role, stopping short of detailed score-function derivation.
Examples
Visualizes weights assigned to three digit images inside a bag.
What you can do after
Apply it via PyTorch by defining a score network, normalizing across items, and passing it to a combiner.
How it connects
Connects feature extraction, Softmax, weighted averages, backbone networks, and downstream classification.
Cover of Inside Deep Learning: Math, Algorithms, Models
liveBookmanning.com
Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.