Attention lets a neural network focus on the parts of an input sequence that contribute most to a prediction instead of relying only on one condensed recurrent-state summary. An attention layer learns weights for time steps and combines their features into an attention-weighted context vector. section 4.3.4
Choosing relevant evidence
In “Georgia draws so well, I am a fan of ___ paintings,” the model should emphasize “Georgia” to infer “her.” In “I am a fan of her ____,” it should emphasize “draws” to predict “drawings” or “paintings.” The same principle can focus on selected pixels for image classification or relevant text for phishing-fraud detection. section 4.3.4 section 4.4
Attention-based architectures progress from recurrent models to models built entirely from attention:
- An attention-based recurrent network adds an attention layer above a recurrent layer, learns weights for each time step, and passes the resulting context vector to the output layer. section 4.3.4
- A transformer removes recurrent layers and uses attention layers throughout the architecture, including multi-head attention. section 4.3.4
- Self-attention produces coefficients that assign different importance to input positions; multi-head attention combines several such views so the model can emphasize different parts in different proportions. section 4.3.4
- Graph Attention Networks aggregate neighborhood information with attention-weighted rather than equal weights, allowing connected nodes to receive different treatment. section 9.5
Figure 4.20 Self-attention layer showing the attention weights (Pq, Pk, and Pv) that produce attention coefficients (c1, c2, …, cn) that determine the weightage to be given to particular parts (time-steps) of the input sequence while making a prediction at the output layer.
Trace how self-attention weights become coefficients that determine the importance of sequence positions. section 4.3.4
Figure 4.21 A multi-head attention layer showing the concatenation of features produced by multiple self-attention layers, producing a set of attention coefficients that inform the model to look at different parts of the input sequence in different proportions.
Compare how multiple self-attention layers combine features and attend to different input regions. section 4.3.4







































