Attention contextualizes token representations by weighting information from other tokens in a sequence. In a Transformer block, it works alongside the MLP: attention supplies context, while the MLP processes knowledge. During inference, attention’s compute and dynamic memory demands grow with context length; its key-value (KV) cache can exhaust GPU memory before a long document is finished. section 3.1.2 section 8.0 section 3.0
For each token, a query (Q) is compared with keys (K) using dot products. Softmax normalizes the scores into weights that determine each value’s (V) contribution to the contextualized representation. Multiple attention heads perform this operation in parallel, and their outputs are concatenated. Head specialization emerges during training; illustrative labels should not be treated as directly interpretable. section 3.1.2
Figure 3.4 Attention mechanism for the token "spring." The Query (Q) from "spring" compares with the Keys (K) of all tokens through dot product operations. The resulting scores, after softmax normalization, determine what fraction of each Value (V) contributes to create the enriched embedding.
Trace how the query for one token weights values from the sequence. section 3.1.2
Context and head dimensions
The embedding for “spring” starts with the same token identity in sentences about a mountain spring and a garden in spring. Attention incorporates surrounding words to distinguish the contexts. In DistilGPT2, a 768-dimensional embedding is processed by 12 heads of 64 dimensions each; inspect model.config.n_head and model.config.n_embd to verify the dimensions. section 3.1.2
The KV cache stores keys and values for processed tokens. Multi-head attention keeps a separate KV pair per query head; grouped-query attention (GQA) shares pairs within groups of query heads, and multi-query attention shares one pair across all query heads. Sharing reduces cache storage, making longer contexts possible within the same memory budget. section 3.1.2 section 3.5
Figure 3.5 Attention mechanisms and memory optimization strategies. Multi-Head Attention (MHA) maintains independent K,V pairs for each Q head — one pair per processed token, forming a growing cache. Grouped-Query Attention (GQA) groups Q heads to share K,V pairs (here 6 Q heads in 3 groups, each sharing one K,V pair per token). Multi-Query Attention (MQA) compresses to the extreme: all Q heads share a single K,V pair per token.
Compare how the three attention variants share key-value pairs. section 3.1.2
Two optimization routes target different costs: quantize the KV cache, or identify and remove attention modules that contribute relatively little to the output. Removing an attention module preserves the block’s MLP, unlike removing the entire Transformer block. section 8.0 section 8.3
A practical attention-removal workflow:
- Inspect the model’s structure and choose representative calibration data. Scores depend on the text used for measurement; verify module names rather than assuming they match across architectures. section 8.3.1 section 10.1
- Register PyTorch pre-hooks at input_layernorm and post_attention_layernorm. Compare the captured hidden states with cosine similarity; the latter state includes the attention output and its residual connection. section 8.3.1
- Select low-scoring candidates. For a Llama decoder layer, remove self_attn and input_layernorm together, then replace the forward path so it skips attention but retains the MLP and its residual connection. section 8.4
- Resolve save-and-reload compatibility for the modified architecture, then evaluate speed, memory and generation quality. Excessive removal can cause long-form generation to collapse even when short generation appears viable. section 8.3 section 8.4
Listing 8.6 Registering PyTorch hooks
import torch.nn.functional as F
import numpy as np
from tqdm import tqdm
def setup_attention_hooks(model):
num_layers = len(model.model.layers)
layer_inputs = {} #1
layer_outputs = {} #1
hooks = []
def make_input_hook(layer_idx): #2
def hook(module, args):
layer_inputs[layer_idx] = args[0].detach() #3
return hook
def make_output_hook(layer_idx): #2
def hook(module, args):
layer_outputs[layer_idx] = args[0].detach() #4
return hook
for i, layer in enumerate(model.model.layers):
hooks.append(
layer.input_layernorm.register_forward_pre_hook(
make_input_hook(i) #5
)
)
hooks.append(
layer.post_attention_layernorm.register_forward_pre_hook(
make_output_hook(i) #5
)
)
return hooks, layer_inputs, layer_outputs, num_layers
Inspect the hook registration used to capture states around attention. section 8.3.1
Calibration-dependent candidates
On Cosmopedia, the lowest-scoring attention modules in Llama-3.2-3B were layers 21, 22 and 23. The comparison with Llama-3.1-8B also placed the least important layers between 60% and 80% of network depth, but these rankings depend on the calibration data and require validation after removal. section 8.3.2 section 8.3.1







































