liveBook · Manning

Chapter 8: Attention optimization

What you will learn in chapter 8 of Rearchitecting LLMs

Introduces attention as an inference bottleneck whose costs grow with context length, motivating the chapter’s optimization work.

After this chapter you will be able to

What it builds on

Section by section

Introduction

Introduces attention as an inference bottleneck whose costs grow with context length, motivating the chapter’s optimization work.

Key terms: attention mechanism, computational demand, dynamic memory consumption, context length, batch size

8.1 KV cache fundamentals and Hugging Face implementation

The section introduces KV cache quantization as a way to reduce context-dependent GPU memory use and notes that its behavior can vary across GPU architectures.

Key terms: KV cache, quantization, dynamic memory consumption, context, GPU architecture, quantization type, T4, L4

8.1.1 What the KV cache is and why it matters

The section explains why the KV cache remains necessary during generation and how quantizing its stored tensors reduces memory use, with precision and possible speed trade-offs.

Key terms: KV cache, GQA, attention heads, K and V tensors, FP16, BF16, FP8, 4-bit, dynamic memory consumption, precision

8.1.2 Hugging Face experiment setup

This section sets up a Hugging Face experiment that compares weight and KV-cache quantization and explains how to enable and measure a 4-bit HQQ KV cache during generation.

Key terms: weight quantization, KV cache quantization, BitsAndBytes, HQQ, QuantizedCacheConfig, Key and Value tensors, T4, measure_memory_allocation, cache_config, cache_implementation

8.1.3 Hugging Face results analysis

This section compares memory, throughput, and generated-text quality for a base model, four-bit quantized weights, and a quantized KV cache on T4 and L4 GPUs.

Key terms: static VRAM, Dynamic VRAM, quantized KV cache, HQQ kernel, throughput, L4 GPU, T4 GPU, BitsAndBytes, native quantization capabilities, generated_text

8.2 Scaling KV cache for production with vLLM

This section explains how vLLM’s VRAM allocation affects KV-cache capacity and how the accompanying notebooks run vLLM outside Jupyter.

Key terms: vLLM, PagedAttention, VRAM, KV tokens, KV cache, blocks, context capacity, batch size, fileno(), stdout

8.2.1 vLLM experiment configuration

This section configures vLLM experiments to compare KV-cache and model-weight quantization while recording cache capacity and generation throughput.

Key terms: run_vllm_test.py, -fp8, -nf4, KV tokens metric, throughput, FP8, 4-bit tensors, latency, model weights, static VRAM

8.2.2 vLLM results analysis

This section compares vLLM KV-cache capacity, generation throughput, and output quality for baseline, FP8 KV-cache, and NF4 weight configurations on T4 and L4 GPUs.

Key terms: KV tokens, capacity gain, baseline, FP8, KV cache, T4, L4, FP16, throughput, NF4 weights

8.3 Measuring attention redundancy

The section shifts redundancy analysis from whole transformer blocks to attention layers and explains why attention can be removed while retaining the MLP.

Key terms: PyTorch hooks, cosine distance, attention layer, MLP layer, redundancy

8.3.1 Scoring with cosine similarity

This section shows how to measure each attention module’s contribution by capturing activations around attention, scoring their difference with cosine similarity, and using representative calibration text.

Key terms: input_layernorm, post_attention_layernorm, register_forward_pre_hook, X_A, Y_A, residual connection, calculate_cosine_importance, cosine similarity, attention mask, importance score

8.3.2 Analyzing results

This section interprets attention-module importance scores across two Llama models to locate pruning candidates and compare how much low-scoring capacity each model has.

Key terms: importance scores, Attention Module, pruning candidates, Cosmopedia dataset, network depth, threshold, redundancy space

8.4 Removing attention modules

This section selects low-scoring decoder blocks for attention removal and shows how to remove their attention phase while keeping the model's forward pass functional.

Key terms: ascending score, layers_to_drop, redundancy space, input_layernorm, self_attn, post_attention_layernorm, mlp, delattr, forward(), forward_no_attn

8.4.1 Structural implications and framework compatibility

Explains why an attention-pruned model cannot be reloaded with the standard architecture and how a custom class restores the pruned structure before checkpoint weights are loaded.

Key terms: save_pretrained, LlamaForCausalLM, missing keys, config.json, dropped_attn_layers, auto_map, PrunedLlamaForCausalLM, trust_remote_code=True, _patch_layer, forward()

8.4.2 Memory savings, quality trade-offs, and pruning limits

This section compares memory savings and quality losses at three attention-pruning levels, then shows how distillation can recover generation quality after moderate pruning.

Key terms: Lambada, multiple-choice or binary-selection benchmarks, open-vocabulary generation, long-range contextual integration, KV cache, dynamic VRAM, peak GPU memory growth, static benchmarks, autoregressively, repetition loop

8.5 From paper to practice

This section connects attention-module pruning to research showing why redundancy scores must include the residual connection.

Key terms: transformer block, components, cosine similarity, residual connection, input_layernorm, post_attention_layernorm, X_A, Y_A

8.5.1 Pruning strategies and evaluation at scale

The section compares four pruning strategies and uses performance, speedup, and model-scale results to show why attention removal can be effective but must be evaluated carefully.

Key terms: Block Drop, MLP Drop, Attention Drop, Joint Layer Drop, SDR, Speedup Degradation Ratio, Llama-2-70B, Llama-3.2-3B, attention layers, throughput

8.5.2 Beyond the benchmarks

This section shows why attention-pruning results must be evaluated against the intended task, while noting that attention removal can be combined with quantization.

Key terms: static evaluation, Lambada, autoregressive token-by-token generation, Attention Drop, recovery process, autoregressive coherence, AWQ, 4-bit precision, KV cache quantization, dense retrieval model

8.6 Hands-on lab

8.7 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 7: Model specialization Chapter 9: Dynamic routing with Mixture of Experts →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.