liveBook · Manning

Attention head

As explained in Timeless Algorithms, The Foundational Ideas

A component of an attention architecture that evaluates relationships among sequence elements, potentially focusing on a different type of relationship from other heads.

An attention head evaluates relationships among sequence elements and contributes to contextual representations. Multiple heads can form different views of the same context, representing several kinds of relationships at once. This distributes context across token representations rather than relying solely on one recurrent state, but finite vector dimensions and context-window limits remain. section 12.3.2 section 12.2.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines heads as distinct views of context, without detailing head-specific computations or parameters.
Examples
Uses the trophy–box–it pronoun example to illustrate attention’s contextual relationship weighting.
What you can do after
Understand how multiple attention heads represent several kinds of contextual relationships at once.
How it connects
Connects attention heads to self-attention, token representations, recurrent state, and context windows.

Also explained in

Compare how 7 books explain attention head →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.