Attention lets a model selectively focus on relevant portions of its input while producing an output. It represents relevance with weights, helping preserve context across long sequences and distinguish important graph neighbors. section 11.3.2
Attention supports several related operations:
- Assigns importance scores to input segments or relationships between graph nodes. section 11.3.2
- Dynamically concentrates computational resources on different parts of the input at different time steps. section 11.3.2
- Improves on graph convolution that applies the same learned transformation to every neighbor, allowing relevant neighbors to receive different weights. section 11.3.2
- Provides the foundation for Transformer processing, which addresses loss of early context in long sequential inputs. section 11.3.2
Figure 11.9 Graph convolutional network (GCN) vs. graph attention network (GAT)
Compare graph convolutional and graph attention networks to see how attention makes neighbor importance explicit. section 11.3.2
Pointer networks and convex hulls
A pointer network uses attention as a pointer that selects an element from a variable-sized input sequence for the output. For a convex hull, it can select points defining a polygon that encloses all input points while minimizing the boundary; an example output sequence is [1, 4, 2] from [1, 2, 3, 4]. section 11.3.3
