liveBook · Manning

Attention coefficient

As explained in Optimization Algorithms: AI techniques for design, planning, and control problems

A numerical factor that quantifies the importance assigned to one node or input element relative to another.

An attention coefficient is a numerical weighting factor that indicates how strongly one input element or graph node should influence another. In a graph attention network, αvu weights node u when node v aggregates information, replacing the unweighted adjacency used by a graph convolutional network. section 11.3.2

Attention coefficients are computed and used through a short sequence:

  • Calculate an unnormalized score between node features, using dot-product similarity or a learned attention mechanism. section 11.3.2
  • Restrict attention to the node’s neighborhood rather than the entire graph. section 11.3.2
  • Normalize the scores with softmax so coefficients remain comparable across neighborhoods. section 11.3.2

Multiple attention heads

With multi-head attention, different sets of coefficients can focus on different parts of the input. In the illustrated three-head example, node 5 uses α52, α54, and α55; the resulting head-specific embeddings are averaged. section 11.3.2

How Optimization Algorithms: AI techniques for design, planning, and control problems covers it

How deep it goes
Explains scoring, neighborhood restriction, softmax normalization, and multi-head use without deriving implementation equations.
Examples
Uses neighboring graph nodes and a three-head example in which node 5 receives coefficients α52, α54, and α55.
What it assumes
Assumes familiarity with node features, graph neighborhoods, embeddings, and basic softmax normalization.
What you can do after
Evaluate how graph attention weights emphasize particular neighbors during feature aggregation.
How it connects
Connects graph attention networks with graph convolutional networks, dot-product attention, softmax, and Transformers.
Cover of Optimization Algorithms: AI techniques for design, planning, and control problems
liveBookmanning.com
Read Optimization Algorithms: AI techniques for design, planning, and control problems on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Optimization Algorithms: AI techniques for design, planning, and control problems. The section links point to the parts of the book it draws on.