Chapter 8: Attention optimization
Introduces attention as an inference bottleneck whose costs grow with context length, motivating the chapter’s optimization work.
After this chapter you will be able to
- Attention inference bottleneck. Explain why attention limits LLM inference as context length increases.
- KV cache quantization benefits. Explain why quantizing the KV cache can help a model process larger prompts: the cache grows with context, and reducing its precision lowers dynamic memory consumption.
- KV cache necessity. Explain why disabling the KV cache is not a practical way to save memory during generation.
- Quantized cache write read cycle. Trace what happens to K and V tensors when a quantized KV cache writes them to memory and uses them for a new Query.
- Independent quantization comparison. Explain how the experiment isolates weight quantization from KV-cache quantization by running the same prompt on the same model in three configurations and comparing memory use and throughput.
- Quantized cache activation. Explain how the optional cache configuration changes the arguments passed to model.generate() and why the quantized cache implementation must be specified.
- Tokenwise cache compression. Trace what happens to newly generated Key and Value tensors when HQQ KV-cache quantization is enabled.
- Memory and throughput trade-offs. Explain how the three configurations differ in static VRAM, dynamic VRAM growth, and throughput on the T4 GPU.
- Generated-text quality difference. Explain the observed quality difference between weight and KV cache quantization and the section's proposed reason for it.
- vLLM memory allocation. Explain that vLLM uses PagedAttention and pre-allocates a fixed percentage of available VRAM at startup, so reducing model-weight memory leaves room for more KV tokens.
- KV cache capacity metric. Explain why the total number of KV tokens the engine can handle is the relevant metric when quantizing the KV cache, and relate more memory blocks to supported context capacity and batch size.
- vLLM experiment controls and measurements. Describe what the script’s two optional flags change and how it records results for comparison.
- FP8 cache precision rationale. Explain why this vLLM experiment does not use a 4-bit KV cache as the earlier Hugging Face experiment did.
- Weight quantization and cache space. Explain how quantizing model weights changes the VRAM available to vLLM’s KV cache in the second experiment.
- FP8 Hardware Trade-off. Explain why FP8 KV-cache quantization roughly doubles capacity on both GPUs but slows generation on the T4 and not on the L4.
- NF4 Throughput Advantage. Explain why NF4 weight quantization raised throughput on both GPUs while giving a smaller KV-token capacity gain than FP8.
- Output Quality Qualification. Describe how the generated text changes the interpretation of the capacity and throughput results, including what happened with NF4 on both GPUs.
- Attention-level redundancy assessment. Explain how this approach adapts depth-pruning analysis to examine attention layers rather than whole transformer blocks, and why removing a redundant attention layer would leave the block’s MLP in place.
- Attention phase hook placement. Explain how two hooks isolate the attention phase rather than measuring the whole Transformer block, and describe the tensors captured at each point.
- Cosine attention importance score. Describe how captured activations become an attention-importance score and how scores are combined across batches.
- Attention importance across layers. Describe where attention-module importance scores are high and low in Llama-3.2-3B, and relate the high scores at the beginning and end to the roles stated for those layers.
- Redundancy across model sizes. Compare the location and proportion of low-scoring attention modules in the 3B and 8B models, including what the section concludes about pruning larger models.
- Low-score attention selection. Explain how attention-removal candidates are chosen by sorting scores in ascending order and why the example removes only three modules.
- Paired attention-phase removal. Describe which components of a decoder block are deleted together, which remain, and why the pair is removed.
- Functional forward-pass replacement. Trace why deleting the modules requires a replacement forward method and how the replacement keeps the pruned block running.
- Standard reload failure. Explain why reloading the pruned model with the original LlamaForCausalLM class does not preserve the pruning, even though loading completes.
- Configuration and custom class routing. Describe how the configuration records modified layers and directs Hugging Face to instantiate the custom model class on reload.
- Patching before weight loading. Trace how the custom constructor rebuilds the pruned architecture before checkpoint weights are loaded, and explain why this avoids missing or unexpected keys.
- Benchmark sensitivity to pruning. Explain why removing three or six attention layers leaves most benchmark scores nearly unchanged while Lambada declines, and describe what happens at 15 removed layers.
- Dynamic memory savings. Explain how removing attention layers reduces dynamic VRAM during generation and describe how the savings change across the pruning configurations.
- Generation quality and benchmark limits. Describe how generated text deteriorates at three, six, and 15 removed layers, and explain why static selection benchmarks can miss that deterioration.
- Distillation-based quality recovery. Explain what happened when short knowledge distillation was applied after six attention layers were removed, including the roles of its two loss components.
- Residual-aware similarity measurement. Explain how the paper measures module redundancy with cosine similarity and why the residual connection must be included; describe where the notebook captures the attention input and output.
- Four pruning strategies. Distinguish Block Drop, MLP Drop, Attention Drop, and Joint Layer Drop by what each removes. Explain why treating a whole block as one unit can miss differences in redundancy between its attention and MLP modules.
- Speedup Degradation Ratio. Explain how SDR compares pruning strategies and what a low value means. Describe the section's finding that Attention Drop has a much lower SDR than Block Drop or MLP Drop.
- Model scale and pruning limits. Explain how the reported tolerance for attention removal differs between the large and small models. Use the Llama-2-70B and Llama-3.2-3B results to show why pruning ratios must be checked against both speed and generation quality.
- Static benchmark blind spots. Explain why multiple-choice and binary classification benchmarks can miss degradation in free-form generation after attention layers are removed. Describe what the added Lambada and text-generation tests revealed.
- Task-dependent recovery decision. Distinguish when Attention Drop without recovery is viable from when a recovery process is needed, based on the model's intended task.
- Task-shaped architectural redundancy. Explain how findings from dense retrieval contrast with the attention-pruning findings and what that contrast implies for choosing an architecture.
What it builds on
- Quantization and the memory bottleneck (chapter 7) · Uses the earlier idea of quantization to reduce memory use.
- Optimized attention: from Multi-Head Attention (MHA) to Grouped-Query Attention (GQA) (chapter 3) · Introduces GQA and its relationship to attention heads.
- Classical attention mechanism (chapter 3) · Introduces attention and its Query, Key, and Value tensors.
- Using PyTorch hooks (chapter 4) · Earlier use of PyTorch hooks to examine transformer blocks.
- Understanding cosine similarity (chapter 4) · Earlier use of cosine-based comparison to assess contributions.
- General behaviour of a Transformer model (chapter 3) · The general flow of computation through transformer blocks.
- The compound loss function (chapter 6) · Assumes familiarity with the ground-truth and teacher-alignment components of the distillation loss.
- (chapter intro) (chapter 5) · The distinction between pruning model width and removing other components.
Section by section
Introduction
Introduces attention as an inference bottleneck whose costs grow with context length, motivating the chapter’s optimization work.
- Attention inference bottleneck: Explain why attention limits LLM inference as context length increases.
Why it matters: This bottleneck explains why reducing attention costs matters for inference.
Key terms: attention mechanism, computational demand, dynamic memory consumption, context length, batch size
8.1 KV cache fundamentals and Hugging Face implementation
The section introduces KV cache quantization as a way to reduce context-dependent GPU memory use and notes that its behavior can vary across GPU architectures.
- KV cache quantization benefits: Explain why quantizing the KV cache can help a model process larger prompts: the cache grows with context, and reducing its precision lowers dynamic memory consumption.
Why it matters: It explains why cache precision matters when prompt length pushes GPU memory limits. - GPU-dependent quantization behavior: State why GPU architecture matters when assessing quantization: the same quantization type can behave differently on different GPUs.
Why it matters: A result observed on one GPU may not describe how the same quantization behaves on another.
Key terms: KV cache, quantization, dynamic memory consumption, context, GPU architecture, quantization type, T4, L4
8.1.1 What the KV cache is and why it matters
The section explains why the KV cache remains necessary during generation and how quantizing its stored tensors reduces memory use, with precision and possible speed trade-offs.
- KV cache necessity: Explain why disabling the KV cache is not a practical way to save memory during generation.
Why it matters: It explains why the section focuses on shrinking the cache rather than removing it. - Quantized cache write read cycle: Trace what happens to K and V tensors when a quantized KV cache writes them to memory and uses them for a new Query.
Why it matters: The write-read cycle shows where cache memory is saved without giving up cached K and V tensors. - Cache quantization trade-offs: Describe the costs that can accompany the memory savings from KV-cache quantization.
Why it matters: Smaller cache storage is not automatically a faster or cost-free inference setup.
Key terms: KV cache, GQA, attention heads, K and V tensors, FP16, BF16, FP8, 4-bit, dynamic memory consumption, precision
8.1.2 Hugging Face experiment setup
This section sets up a Hugging Face experiment that compares weight and KV-cache quantization and explains how to enable and measure a 4-bit HQQ KV cache during generation.
- Independent quantization comparison: Explain how the experiment isolates weight quantization from KV-cache quantization by running the same prompt on the same model in three configurations and comparing memory use and throughput.
Why it matters: Keeping the two changes separate makes their effects on GPU memory easier to interpret. - HQQ cache backend choice: Justify the choice of HQQ for KV-cache compression and describe the cache configuration used in the experiment.
Why it matters: The cache configuration depends on a backend that can perform compression in the experiment's GPU environment. - Quantized cache activation: Explain how the optional cache configuration changes the arguments passed to model.generate() and why the quantized cache implementation must be specified.
Why it matters: A cache configuration alone does not enable compression; generation must be told to use the quantized cache. - Tokenwise cache compression: Trace what happens to newly generated Key and Value tensors when HQQ KV-cache quantization is enabled.
Why it matters: This explains what is compressed during generation and when the cached values are restored for attention.
Key terms: weight quantization, KV cache quantization, BitsAndBytes, HQQ, QuantizedCacheConfig, Key and Value tensors, T4, measure_memory_allocation, cache_config, cache_implementation
8.1.3 Hugging Face results analysis
This section compares memory, throughput, and generated-text quality for a base model, four-bit quantized weights, and a quantized KV cache on T4 and L4 GPUs.
- Memory and throughput trade-offs: Explain how the three configurations differ in static VRAM, dynamic VRAM growth, and throughput on the T4 GPU.
Why it matters: The memory benefit depends on what is quantized and which kind of memory is measured. - T4 and L4 differences: Describe which memory results carried over to the L4 GPU and what differed for KV cache and weight quantization.
Why it matters: Results on a newer GPU need not match every T4 result or improve every quantized configuration. - Generated-text quality difference: Explain the observed quality difference between weight and KV cache quantization and the section's proposed reason for it.
Why it matters: Lower memory use is not a useful win if the generated response breaks down.
Key terms: static VRAM, Dynamic VRAM, quantized KV cache, HQQ kernel, throughput, L4 GPU, T4 GPU, BitsAndBytes, native quantization capabilities, generated_text
8.2 Scaling KV cache for production with vLLM
This section explains how vLLM’s VRAM allocation affects KV-cache capacity and how the accompanying notebooks run vLLM outside Jupyter.
- vLLM memory allocation: Explain that vLLM uses PagedAttention and pre-allocates a fixed percentage of available VRAM at startup, so reducing model-weight memory leaves room for more KV tokens.
Why it matters: It explains why shrinking model weights can increase serving capacity in vLLM. - KV cache capacity metric: Explain why the total number of KV tokens the engine can handle is the relevant metric when quantizing the KV cache, and relate more memory blocks to supported context capacity and batch size.
Why it matters: KV-token capacity connects a memory-saving strategy to the workloads the server can support. - Notebook initialization workaround: Describe why initializing vLLM in Jupyter or Colab can raise an UnsupportedOperation error and how the notebook runs it in a standalone Python process instead.
Why it matters: It helps explain the notebook’s execution pattern if vLLM fails to initialize directly in a cell.
Key terms: vLLM, PagedAttention, VRAM, KV tokens, KV cache, blocks, context capacity, batch size, fileno(), stdout
8.2.1 vLLM experiment configuration
This section configures vLLM experiments to compare KV-cache and model-weight quantization while recording cache capacity and generation throughput.
- vLLM experiment controls and measurements: Describe what the script’s two optional flags change and how it records results for comparison.
Why it matters: These controls and measurements make the vLLM configurations comparable. - FP8 cache precision rationale: Explain why this vLLM experiment does not use a 4-bit KV cache as the earlier Hugging Face experiment did.
Why it matters: Smaller cache values do not automatically make a throughput-focused deployment faster. - Weight quantization and cache space: Explain how quantizing model weights changes the VRAM available to vLLM’s KV cache in the second experiment.
Why it matters: Weight quantization can increase cache capacity even though it does not directly quantize the cache.
Key terms: run_vllm_test.py, -fp8, -nf4, KV tokens metric, throughput, FP8, 4-bit tensors, latency, model weights, static VRAM
8.2.2 vLLM results analysis
This section compares vLLM KV-cache capacity, generation throughput, and output quality for baseline, FP8 KV-cache, and NF4 weight configurations on T4 and L4 GPUs.
- KV Token Capacity Metric: Explain that the vLLM results report how many KV tokens the engine can hold, rather than memory use in megabytes, and state how the percentage capacity gain is calculated against the baseline.
Why it matters: Using the right metric makes the capacity comparisons meaningful. - FP8 Hardware Trade-off: Explain why FP8 KV-cache quantization roughly doubles capacity on both GPUs but slows generation on the T4 and not on the L4.
Why it matters: The same cache optimization can have a different speed trade-off on different GPUs. - NF4 Throughput Advantage: Explain why NF4 weight quantization raised throughput on both GPUs while giving a smaller KV-token capacity gain than FP8.
Why it matters: It explains why shrinking weights can improve generation speed more than shrinking the KV cache. - Output Quality Qualification: Describe how the generated text changes the interpretation of the capacity and throughput results, including what happened with NF4 on both GPUs.
Why it matters: A faster configuration is not useful on these results if its responses degrade.
Key terms: KV tokens, capacity gain, baseline, FP8, KV cache, T4, L4, FP16, throughput, NF4 weights
8.3 Measuring attention redundancy
The section shifts redundancy analysis from whole transformer blocks to attention layers and explains why attention can be removed while retaining the MLP.
- Attention-level redundancy assessment: Explain how this approach adapts depth-pruning analysis to examine attention layers rather than whole transformer blocks, and why removing a redundant attention layer would leave the block’s MLP in place.
Why it matters: It explains why attention can be assessed and pruned separately instead of discarding an entire transformer block.
Key terms: PyTorch hooks, cosine distance, attention layer, MLP layer, redundancy
8.3.1 Scoring with cosine similarity
This section shows how to measure each attention module’s contribution by capturing activations around attention, scoring their difference with cosine similarity, and using representative calibration text.
- Attention phase hook placement: Explain how two hooks isolate the attention phase rather than measuring the whole Transformer block, and describe the tensors captured at each point.
Why it matters: The capture points determine whether the score describes attention alone or the entire block. - Cosine attention importance score: Describe how captured activations become an attention-importance score and how scores are combined across batches.
Why it matters: The score turns captured changes in hidden states into a comparable measure for attention modules. - Task matched calibration data: Justify why the calibration text spans multiple kinds of tasks rather than a single domain.
Why it matters: A score measured on unrepresentative text may not reflect the work the optimized model is meant to do.
Key terms: input_layernorm, post_attention_layernorm, register_forward_pre_hook, X_A, Y_A, residual connection, calculate_cosine_importance, cosine similarity, attention mask, importance score
8.3.2 Analyzing results
This section interprets attention-module importance scores across two Llama models to locate pruning candidates and compare how much low-scoring capacity each model has.
- Attention importance across layers: Describe where attention-module importance scores are high and low in Llama-3.2-3B, and relate the high scores at the beginning and end to the roles stated for those layers.
Why it matters: The score pattern identifies candidate attention modules without treating every layer as equally expendable. - Redundancy across model sizes: Compare the location and proportion of low-scoring attention modules in the 3B and 8B models, including what the section concludes about pruning larger models.
Why it matters: The comparison explains why the section expects more room for attention pruning in the larger model.
Key terms: importance scores, Attention Module, pruning candidates, Cosmopedia dataset, network depth, threshold, redundancy space
8.4 Removing attention modules
This section selects low-scoring decoder blocks for attention removal and shows how to remove their attention phase while keeping the model's forward pass functional.
- Low-score attention selection: Explain how attention-removal candidates are chosen by sorting scores in ascending order and why the example removes only three modules.
Why it matters: Selection uses the measured scores, but the amount removed must also reflect the model's limited pruning tolerance. - Paired attention-phase removal: Describe which components of a decoder block are deleted together, which remain, and why the pair is removed.
Why it matters: Removing the whole attention phase avoids retaining work that no longer serves a purpose. - Functional forward-pass replacement: Trace why deleting the modules requires a replacement forward method and how the replacement keeps the pruned block running.
Why it matters: Deleting parameters is not enough: execution must also stop referring to the missing modules.
Key terms: ascending score, layers_to_drop, redundancy space, input_layernorm, self_attn, post_attention_layernorm, mlp, delattr, forward(), forward_no_attn
8.4.1 Structural implications and framework compatibility
Explains why an attention-pruned model cannot be reloaded with the standard architecture and how a custom class restores the pruned structure before checkpoint weights are loaded.
- Standard reload failure: Explain why reloading the pruned model with the original LlamaForCausalLM class does not preserve the pruning, even though loading completes.
Why it matters: A successful load does not guarantee that a structurally pruned model has been restored correctly. - Configuration and custom class routing: Describe how the configuration records modified layers and directs Hugging Face to instantiate the custom model class on reload.
Why it matters: The loader needs both the layer indices and the right class to reconstruct the modified model. - Patching before weight loading: Trace how the custom constructor rebuilds the pruned architecture before checkpoint weights are loaded, and explain why this avoids missing or unexpected keys.
Why it matters: Matching the model skeleton to the checkpoint before loading prevents the removed attention modules from being randomly recreated.
Key terms: save_pretrained, LlamaForCausalLM, missing keys, config.json, dropped_attn_layers, auto_map, PrunedLlamaForCausalLM, trust_remote_code=True, _patch_layer, forward()
8.4.2 Memory savings, quality trade-offs, and pruning limits
This section compares memory savings and quality losses at three attention-pruning levels, then shows how distillation can recover generation quality after moderate pruning.
- Benchmark sensitivity to pruning: Explain why removing three or six attention layers leaves most benchmark scores nearly unchanged while Lambada declines, and describe what happens at 15 removed layers.
Why it matters: A nearly unchanged aggregate of selection-task scores can conceal a meaningful loss of contextual ability. - Dynamic memory savings: Explain how removing attention layers reduces dynamic VRAM during generation and describe how the savings change across the pruning configurations.
Why it matters: Dynamic VRAM reduction is the principal practical benefit identified here for the 3B model on constrained hardware. - Generation quality and benchmark limits: Describe how generated text deteriorates at three, six, and 15 removed layers, and explain why static selection benchmarks can miss that deterioration.
Why it matters: Testing actual generation can reveal pruning limits that strong static benchmark scores obscure. - Speed claims and trade-offs: State how the section qualifies its throughput measurements and explain what a low Speedup Degradation Ratio means.
Why it matters: Memory savings and apparent throughput changes should not be mistaken for a reliable speedup at acceptable quality. - Distillation-based quality recovery: Explain what happened when short knowledge distillation was applied after six attention layers were removed, including the roles of its two loss components.
Why it matters: The six-layer generation failure was not necessarily the final quality limit of that pruned architecture.
Key terms: Lambada, multiple-choice or binary-selection benchmarks, open-vocabulary generation, long-range contextual integration, KV cache, dynamic VRAM, peak GPU memory growth, static benchmarks, autoregressively, repetition loop
8.5 From paper to practice
This section connects attention-module pruning to research showing why redundancy scores must include the residual connection.
- Component-level pruning premise: Explain why a transformer block’s less important components might be removed instead of removing the whole block.
Why it matters: This is the reason to assess attention modules individually rather than treat each block as indivisible. - Residual-aware similarity measurement: Explain how the paper measures module redundancy with cosine similarity and why the residual connection must be included; describe where the notebook captures the attention input and output.
Why it matters: Leaving out the residual can make a pruning score look useful even though removing a few layers substantially harms performance.
Key terms: transformer block, components, cosine similarity, residual connection, input_layernorm, post_attention_layernorm, X_A, Y_A
8.5.1 Pruning strategies and evaluation at scale
The section compares four pruning strategies and uses performance, speedup, and model-scale results to show why attention removal can be effective but must be evaluated carefully.
- Four pruning strategies: Distinguish Block Drop, MLP Drop, Attention Drop, and Joint Layer Drop by what each removes. Explain why treating a whole block as one unit can miss differences in redundancy between its attention and MLP modules.
Why it matters: What gets removed matters as much as how much gets removed. - Speedup Degradation Ratio: Explain how SDR compares pruning strategies and what a low value means. Describe the section's finding that Attention Drop has a much lower SDR than Block Drop or MLP Drop.
Why it matters: SDR makes the speed-versus-quality trade-off explicit rather than judging pruning by speed alone. - Model scale and pruning limits: Explain how the reported tolerance for attention removal differs between the large and small models. Use the Llama-2-70B and Llama-3.2-3B results to show why pruning ratios must be checked against both speed and generation quality.
Why it matters: A pruning ratio that works well for a large model may offer no useful gain—or destroy output quality—in a smaller one.
Key terms: Block Drop, MLP Drop, Attention Drop, Joint Layer Drop, SDR, Speedup Degradation Ratio, Llama-2-70B, Llama-3.2-3B, attention layers, throughput
8.5.2 Beyond the benchmarks
This section shows why attention-pruning results must be evaluated against the intended task, while noting that attention removal can be combined with quantization.
- Static benchmark blind spots: Explain why multiple-choice and binary classification benchmarks can miss degradation in free-form generation after attention layers are removed. Describe what the added Lambada and text-generation tests revealed.
Why it matters: A benchmark can look reassuring while missing a failure that matters in the model's actual use. - Task-dependent recovery decision: Distinguish when Attention Drop without recovery is viable from when a recovery process is needed, based on the model's intended task.
Why it matters: The decision to recover a pruned model depends on what users will ask it to do. - Quantization and attention removal: Explain why attention layer removal and quantization can be applied together rather than treated as alternatives.
Why it matters: Combining compatible optimizations can yield benefits from both rather than forcing a choice between them. - Task-shaped architectural redundancy: Explain how findings from dense retrieval contrast with the attention-pruning findings and what that contrast implies for choosing an architecture.
Why it matters: A pruning result should not be generalized to a different task whose metrics reward different model behavior.
Key terms: static evaluation, Lambada, autoregressive token-by-token generation, Attention Drop, recovery process, autoregressive coherence, AWQ, 4-bit precision, KV cache quantization, dense retrieval model
8.6 Hands-on lab
8.7 Summary
Where this is used later
- The serving problem for a rearchitected model (chapter 12) · Structural implications and framework compatibility of removing attention modules.
- Will it load: from research artifact to servable model (chapter 12) · Removing attention modules has structural implications.
Concepts explained in this chapter
- kv cache quantizationKV cache quantization reduces the numerical precision of stored attention key and value states to lower memory use during autoregressive inference.
- Attention DropAttention Drop is a training-free technique that removes attention layers from a transformer model to reduce computational or memory requirements.
- recoveryA recovery process is a post-restructuring procedure that restores or improves a model’s performance after substantial structural components have been removed.
- attentionAttention is a neural-network mechanism that computes input-dependent weights to prioritize information during processing.
- attention pruningAttention pruning removes or bypasses attention components in a neural model to reduce resource requirements.
- key-value cacheA key-value cache stores previously computed attention key and value tensors to accelerate autoregressive inference.
- LAMBADALAMBADA is a language-modeling benchmark that evaluates whether a model can predict a target word from a textual passage using its preceding context.
- speedup degradation ratioSDR, or Speedup Degradation Ratio, is the percentage of performance drop divided by the percentage of speedup gained from a model modification.
- distillationKnowledge distillation is a model-compression technique in which a smaller student model learns to reproduce useful behavior from a larger teacher model.
- quantizationQuantization represents model parameters or intermediate states with lower-precision numerical formats to reduce storage and computation requirements.
- structural modificationAttention removal is a model modification that removes attention sub-layers from selected transformer blocks.
- memory usageMemory consumption is the amount of computer memory required to store and execute a model or computation.
- layer normalizationLayer normalization standardizes neural-network activations within each representation to stabilize computation.
- Llama-3.2-3BLlama-3.2-3B is a language model in Meta's Llama 3.2 family with approximately three billion parameters.
- generation collapseGeneration collapse is a failure mode in which autoregressive text generation rapidly becomes incoherent, repetitive, or structurally degenerate.
- long-range contextLong-range context is information occurring substantially earlier in a sequence that remains relevant to predicting a later token.
- throughputThroughput is the amount of model work or output that a system completes per unit of time.
- residual connectionA residual connection adds a module's input representation to its transformed output so information can bypass the transformation.
- calibration dataCalibration data is representative input data used to measure model behavior and guide data-dependent model compression decisions.
- repetition loopA repetition loop is a generation failure mode in which a language model repeatedly produces the same text pattern instead of progressing normally.
- pre-hookA pre-hook is a callback that receives a module's input arguments before the module executes.
- register_forward_pre_hookregister_forward_pre_hook is a PyTorch method that registers a callback to run before a module's forward computation.
- vLLMvLLM is an open-source library and inference engine for efficient large language model serving.
- autoregressive generationAutoregressive generation produces a sequence one token at a time, conditioning each new token on previously generated tokens.
- pruningA pruning strategy is a method for deciding which model components to remove during model compression.
- NormalFloat44-bit NormalFloat quantization represents model weights with a four-bit NormalFloat numerical encoding tailored to the distribution of language-model weights.
- feature importance scoreAn importance score numerically estimates how important a model component is to its information or performance.
- neural network layerA model layer is a computational stage in a neural network that transforms incoming representations.
- PagedAttentionPagedAttention is an attention-memory management technique that stores key-value cache blocks in pages to improve memory utilization during inference.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
