liveBook · Manning

Attention

As explained in Build a Large Language Model (From Scratch)

A neural-network mechanism that weights input tokens or values by relevance so the model can focus on important context.

Attention lets a neural network assign different importance to input elements, giving each output access to the context most relevant to its computation. In transformer-based language models, self-attention lets every sequence position interact with and weight other positions. GPT-like models use decoder-only transformer modules and apply attention while generating text from left to right. section 1.8 section 3.2

Build attention progressively through these implementation variants:

  • Simplified self-attention computes attention weights with dot products and forms each context vector as a weighted sum of the inputs. section 3.7
  • Scaled-dot-product self-attention adds trainable weight matrices that transform inputs into queries, keys, and values. section 3.7
  • Causal attention applies a mask so a left-to-right language model cannot access future tokens during generation. section 3.0 section 3.7
  • Multi-head attention stacks multiple causal attention instances, enabling parallel attention to different aspects of the input; batched matrix multiplication provides a compact implementation. section 3.0 section 3.7

Attention transforms input elements into enhanced context-vector representations containing information from the full accessible input. Dot products provide a compact relevance calculation, while matrix multiplication replaces nested loops for efficient computation. Dropout can mask randomly selected attention weights to reduce overfitting. section 3.7

Figure 3.2 The figure depicts different attention mechanisms we will code in this chapter, starting with a simplified version of self-attention before adding the trainable weights. The causal attention mechanism adds a mask to self-attention that allows the LLM to generate one word at a time. Finally, multi-head attention organizes the attention mechanism into multiple heads, allowing the model to capture various aspects of the input data in parallel.
figure

Use this progression diagram to connect simplified self-attention, trainable queries-keys-values attention, causal masking, and multi-head attention. section 3.0

How Build a Large Language Model (From Scratch) covers it

How deep it goes
Explains weighted sums, dot products, queries, keys, values, causal masks, dropout, and multi-head stacking, without formal derivations.
Examples
Demonstrates attention through simplified dot products, weighted context vectors, causal masks, dropout masks, and stacked attention modules.
What it assumes
Assumes tokenized inputs encoded as embeddings and develops attention mechanisms before assembling the surrounding language model.
What you can do after
Build it from scratch as simplified, causal, and multi-head attention modules for a GPT-like language model.
How it connects
Connects attention with self-attention, transformer architecture, causal masking, dropout, queries, keys, values, and multi-head attention.
Cover of Build a Large Language Model (From Scratch)
liveBookmanning.com
Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

Compare how 28 books explain attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.