liveBook · Manning

Attention

As explained in Building Specialized AI Systems

A neural-network component that assigns different weights to input elements when computing a representation or output.

Attention, specifically self-attention, lets each token incorporate information from other tokens when forming a context-aware representation. It captures long-range dependencies without the sequential decay associated with recurrent networks and supports generation, representation learning, and sequence-to-sequence tasks. section 3.1.1

Self-attention computes a token representation in three practical stages:

  • Compare queries with keys to produce compatibility scores. section 3.1.1
  • Normalize the scores with softmax to obtain attention weights. section 3.1.1
  • Combine values through a weighted sum to produce a context-aware output. section 3.1.1

Within a Transformer block, self-attention operates alongside token embeddings, positional information, feed-forward networks, residual connections, and normalization. This architecture forms representations that can later be specialized through fine-tuning. section 3.1.1 section 3.8

Context-aware token output

For a token sequence, the model compares each token's query with other tokens' keys, converts those comparisons into weights, and blends the corresponding values. Tokens that receive larger weights contribute more strongly to the resulting representation. section 3.1.1

Figure 3.1 Encoder-decoder RNN architecture for language translation. The encoder processes the input sequence sequentially, compressing all information into a single hidden state. The decoder then uses this bottleneck representation to generate the output. A key limitation is that the decoder cannot directly access earlier encoder states. It relies entirely on the compressed final hidden state, which can lead to information loss in longer sequences. Image from Sebastian Raschka’s book Build a Large Language Model (From Scratch) [8].

Use the recurrent encoder-decoder contrast to see why attention helps: recurrent encoding compresses a sequence into one final state, creating an information bottleneck for longer inputs. section 3.1.1

How Building Specialized AI Systems covers it

How deep it goes
Explains queries, keys, values, softmax weights, and weighted outputs, while placing attention within Transformer blocks.
Examples
Demonstrates score calculation, softmax normalization, weighted value aggregation, and the recurrent encoder bottleneck.
What you can do after
Understand how Transformers combine token interactions into context-aware representations for later model specialization.
How it connects
Connects self-attention to Transformers, token embeddings, positional information, feed-forward networks, residual connections, normalization, and fine-tuning.

Also explained in

Compare how 40 books explain attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.