liveBook · Manning

Attention

As explained in Build an Advanced RAG Application (From Scratch)

An attention mechanism weights the importance of different elements in an input sequence when computing a representation for a particular element or output.

Attention is a mechanism for weighting which parts of a sequence matter when processing a particular part or producing an output. In Transformers, self-attention relates positions within a sequence so each token can use contextual relationships rather than relying only on nearby or previously processed tokens. This capacity to consider long-range dependencies is central to contextual language understanding. section 2.0 section 2.5

Unlike recurrent networks, which process words one at a time while carrying a hidden state forward, Transformers process a sequence in parallel and use self-attention to identify relevant relationships across it. Parallel processing improves training and inference efficiency, while direct relationships across distant positions help preserve context that sequential models may lose. section 2.1

Figure 2.1 The strength of the transformer model is in its ability to understand the significance and context of every word in a sentence, unlike RNNs which primarily assess the connection of a word with its immediate neighbor.

Compare parallel contextual attention with the more local, sequential relationship handling associated with RNNs. section 2.1

Self-attention sits inside the Transformer’s encoder and decoder architecture, alongside input embeddings and positional encoding. Input embeddings turn tokens into high-dimensional numerical representations; attention then lets positions use one another’s representations. Multi-head attention supplies a broader view of these relationships, while positional encoding preserves information about token order. section 2.2 section 2.5

Attention takes different roles across Transformer components:

  • Encoder self-attention builds contextual representations from an input sequence and supports semantic and syntactic feature extraction. section 2.2 section 2.5
  • Decoder self-attention connects generated tokens, but masks future positions so next-token prediction can use only earlier outputs. section 2.3.2
  • Encoder-decoder attention lets a decoder focus on relevant encoded input while producing an output sequence. section 2.3.2

Reading a sentence as a whole

For “The quick brown fox jumps over the lazy dog,” a recurrent model advances word by word. A Transformer instead examines the full sentence in parallel, using self-attention to focus on words and relationships relevant to each position. This is especially useful when an early word affects the meaning of a much later one. section 2.1

Attention also creates a systems constraint for embedding models. With a fixed context window, self-attention must compute relationships between every token pair, giving quadratic growth in attention work as sequence length grows. At 8,192 tokens, the cited embedding example requires roughly 67 million attention computations; doubling length to 16,384 raises this to about 268 million. Longer inputs also force more information into the same fixed-size embedding vector. section 6.3.1 section 6.3.2

Figure 6.3 Computational Cost of Context Window Expansion

Use the cost comparison to assess why expanding an embedding model’s context window rapidly increases compute requirements. section 6.3.2

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Defines self-attention and its encoder, decoder, masking, and context-window cost roles without detailing attention calculations.
Examples
Uses a fox-and-dog sentence and 8,192-to-32,768-token embedding context scenarios to illustrate attention behavior and cost.
What it assumes
Introduces tokens and embeddings before placing self-attention within encoder and decoder processing.
What you can do after
Evaluate/understand why attention improves long-range context yet constrains embedding context length through quadratic computation.
How it connects
Connects self-attention with Transformers, encoder-decoder attention, multi-head attention, positional encoding, embeddings, RNNs, and GRUs.

Also explained in

Compare how 40 books explain attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.