liveBook · Manning

Attention

As explained in Fight Fraud with Machine Learning

An attention layer is a neural-network layer that computes attention weights over elements or time steps of its input.

Attention lets a neural network focus on the parts of an input sequence that contribute most to a prediction instead of relying only on one condensed recurrent-state summary. An attention layer learns weights for time steps and combines their features into an attention-weighted context vector. section 4.3.4

Choosing relevant evidence

In “Georgia draws so well, I am a fan of ___ paintings,” the model should emphasize “Georgia” to infer “her.” In “I am a fan of her ____,” it should emphasize “draws” to predict “drawings” or “paintings.” The same principle can focus on selected pixels for image classification or relevant text for phishing-fraud detection. section 4.3.4 section 4.4

Attention-based architectures progress from recurrent models to models built entirely from attention:

  • An attention-based recurrent network adds an attention layer above a recurrent layer, learns weights for each time step, and passes the resulting context vector to the output layer. section 4.3.4
  • A transformer removes recurrent layers and uses attention layers throughout the architecture, including multi-head attention. section 4.3.4
  • Self-attention produces coefficients that assign different importance to input positions; multi-head attention combines several such views so the model can emphasize different parts in different proportions. section 4.3.4
  • Graph Attention Networks aggregate neighborhood information with attention-weighted rather than equal weights, allowing connected nodes to receive different treatment. section 9.5
Figure 4.20 Self-attention layer showing the attention weights (Pq, Pk, and Pv) that produce attention coefficients (c1, c2, …, cn) that determine the weightage to be given to particular parts (time-steps) of the input sequence while making a prediction at the output layer.

Trace how self-attention weights become coefficients that determine the importance of sequence positions. section 4.3.4

Figure 4.21 A multi-head attention layer showing the concatenation of features produced by multiple self-attention layers, producing a set of attention coefficients that inform the model to look at different parts of the input sequence in different proportions.

Compare how multiple self-attention layers combine features and attend to different input regions. section 4.3.4

How Fight Fraud with Machine Learning covers it

How deep it goes
Explains weighted context vectors, self-attention coefficients, multi-head attention, and graph aggregation without detailing training equations.
Examples
Uses sentence completion, dog-versus-cat pixel classification, phishing detection, and Bitcoin transaction graphs as examples.
What you can do after
Evaluate/understand how attention selects relevant sequence positions and supports transformer and graph-based fraud models.
How it connects
Connects recurrent neural networks, transformers, self-attention, multi-head attention, graph neural networks, and Graph Attention Networks.

Also explained in

Compare how 40 books explain attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fight Fraud with Machine Learning. The section links point to the parts of the book it draws on.