A graph attention network (GAT) is a graph neural network that learns how strongly each neighboring node should influence a node’s updated representation. It is useful when importance cannot be inferred from the number of connections alone: a relatively ordinary-looking account or transaction may still be crucial to detecting fraud. section 4.0 section 4.6
A GAT layer replaces equal-weight neighborhood aggregation with learned attention:
- For each connected pair of nodes, transform their features and calculate an attention score. section 4.5.3
- Normalize a node’s scores across its neighbors, then use the resulting weights to combine their features. section 4.5.3
- Optionally use multiple attention heads to learn from several perspectives at once. section 4.5.3
In PyTorch Geometric, `GATConv` adds attention to a convolutional GNN architecture. `GATv2Conv` is a variation intended to address the original layer’s static-attention limitation. A two-layer binary classifier can pair GAT layers with batch normalization, ReLU, dropout, and a sigmoid output. Set `heads` to control multihead attention; the worked classifier uses two heads in its first layer and one in its output layer. section 4.4 section 4.4.1
Listing 4.7 GAT-based architecture
class GAT(torch.nn.Module):
def __init__(self, hidden_layers=32, heads=1, dropout_p=0.0):
super().__init__()
torch.manual_seed(2022)
self.conv1 = GATConv(data.num_node_features,\
hidden_layers, heads, dropout=dropout_p) #1
self.bn1 = nn.BatchNorm1d(hidden_layers*heads) #2
self.conv2 = GATConv(hidden_layers * heads, \
1, dropout=dropout_p)
def forward(self, data, dropout_p=0.0):
x, edge_index = data.x, data.edge_index
x = self.conv1(x, edge_index)
x = self.bn1(x)
x = F.relu(x)
x = F.dropout(x, training=self.training)
x = self.conv2(x, edge_index)
return torch.sigmoid(x)
The PyTorch Geometric model definition shows where attention heads, batch normalization, dropout, and the output layer fit. section 4.4.1
Classifying fraudulent reviews
Represent Yelp reviews as nodes and connect reviews that share characteristics such as an author or business. Convert the graph and features to PyTorch Geometric data, then train a GAT to label each review spam or not spam. Compare it against tabular classifiers and a graph convolutional network (GCN), rather than assuming attention will help. section 4.1 section 4.3.2 section 4.6
Table 4.8 ROC AUC of the models
|
Model
|
ROC AUC (%)
|
|---|---|
|
Logistic regression
|
75.90
|
|
XGBoost
|
94.17
|
|
Multilayer perceptron
|
85.93
|
|
GCN
|
94.37
|
|
GAT
|
95.65
|
|
GATv2
|
95.10
|
The reported ROC AUC comparison puts GAT at 95.65%, GATv2 at 95.10%, GCN at 94.37%, and XGBoost at 94.17% on this review task. section 4.4.1
Attention has a cost: scores for every edge and attention head increase memory use. Use `NeighborLoader` to sample neighbors and train in mini-batches when processing the whole graph is impractical. The review example uses batches of 128 input nodes and samples neighbors over two layers. If training speed or limited compute matters more than integrating relationships, consider XGBoost; also check class imbalance. A snapshot-based review graph does not capture changing behavior over time. section 4.4.1 section 4.4.3 section 4.6 section 6.1
