liveBook · Manning

Attention-based network

As explained in Inside Deep Learning: Math, Algorithms, Models

A neural network that incorporates attention to adapt its computation to the input by focusing on relevant items.

An attention-based network adapts computation to each input by scoring hidden items, selecting relevant information, and predicting from the resulting context. A backbone produces hidden states, while separate scoring, attention, and prediction subnetworks handle the non-sequential computation. section 10.3

A typical forward pass:

  • Compute a masked context by dividing only by the number of valid items, with a small defensive constant for empty bags. section 10.3

Variable-length digit sets

A modified MNIST dataset samples a variable number of digits, pads shorter sets, and assigns the largest sampled digit as the label. An attention model reaches 0.967 variable-length accuracy in the shown evaluation.

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes
Explains the backbone-to-prediction forward pass, masked context calculation, score choices, and defensive padding handling.
Examples
Uses variable-length MNIST digit sets, padded inputs, and dot, general, and additive scoring comparisons.
What it assumes
Assumes familiarity with PyTorch modules, datasets, data loaders, masking, and supervised classification.
What you can do after
Builds and evaluates configurable attention networks in PyTorch for variable-length classification inputs.
How it connects
Connects neural networks with attention mechanisms, masking, context vectors, and classification.
Cover of Inside Deep Learning: Math, Algorithms, Models
liveBookmanning.com
Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.