Chapter 4: Running inference
This chapter introduction previews content generation with SLMs, inference cost estimation, and performance and cost improvements.
After this chapter you will be able to
- Reference model and applicability. Explain why the chapter uses GPT-Neo as its reference model and why its topics are not limited to GPT-Neo.
- GPT-Neo loading and placement. Explain how the section loads GPT-Neo and its tokenizer, and what automatic device placement does when the model does not fit entirely on the GPU.
- Prompt completion workflow. Describe how the section generates readable text completions for a prompt and how the same task can be performed for a batch of prompts.
- Few-shot prompting with examples. Explain that a GPT-based model can use a few examples in a prompt to generalize to unseen examples, such as in text classification. Describe that the generated output includes the input prompt.
- Code Generation Output. Explain that the model can generate code, not just natural language, and describe what the example output contains and why it is truncated.
- Generated Code Reliability. Explain why generated code requires developer review and why an uncustomized pretrained model may produce poor results.
- Predeployment output quality checks. Explain why an LLM’s output should be evaluated before production deployment and what that evaluation helps decide.
- LM-Eval task coverage. Describe what LM-Eval provides for evaluating different models and tasks, including how its flexibility and task versioning help.
- Inference memory estimate. Explain how parameter count, numerical precision, and overhead affect the GPU memory needed to load a model for inference.
- Serving performance measures. Distinguish GPU utilization, time to first token, time per output token, latency, and throughput when assessing LLM serving.
- Metrics for inference improvements. Explain how inference-serving metrics help identify opportunities to reduce costs while making effective use of computational power and keeping users engaged.
- GPU capacity and performance limits. Explain how compute throughput, memory capacity, and memory bandwidth affect the choice and use of a GPU for inference. Describe how the T4's compute throughput and bandwidth yield an ops/byte threshold of 216.6.
- KV cache during generation. Trace the prefill and autoregressive sampling phases and explain why retaining the key–value cache speeds up generation despite using additional memory.
- Attention arithmetic intensity diagnosis. Explain arithmetic intensity as computation divided by bytes accessed, and compare it with the GPU's ops/byte ratio to diagnose the GPT-Neo attention workload on a T4.
- Static and dynamic batching. Distinguish static batching from dynamic batching and explain how running prompts together can improve GPU utilization.
- GPU memory batch limit. Explain how available GPU memory and KV-cache use constrain the number of sequences that fit in a batch, using the GPT-Neo example.
- Throughput and latency tradeoff. Describe when batching is useful and why it may be a poor choice for latency-sensitive applications.
- Per-token bandwidth estimate. Explain how to approximate single-token generation time from model size, parameter precision, and GPU memory bandwidth, using the GPT-Neo example.
- Prefill and total generation time. Describe how prefill changes the total-time estimate and state why the estimate may differ from observed time.
- DeepSpeed inference optimizations. Explain how DeepSpeed can reduce inference memory use and latency through model partitioning and optimized kernels, without requiring model export or new checkpoints for compatible models.
- GPT-Neo inference setup. Describe the section’s GPT-Neo setup: initialize DeepSpeed and CUDA, load the model in half precision, and configure inference with automatic layer replacement and kernel injection. Distinguish the notebook-specific launcher workaround from steps needed outside notebooks.
- Latency and accuracy comparison. Describe how the section compares vanilla and DeepSpeed-optimized GPT-Neo and explain why the measured latency gain must be weighed against output quality.
What it builds on
- The Transformer architecture (chapter 1) · The Transformer attention architecture.
Section by section
Introduction
This chapter introduction previews content generation with SLMs, inference cost estimation, and performance and cost improvements.
4.1 How to generate content
This section introduces GPT-Neo as a reference model and explains why the chapter’s examples extend beyond it.
- Reference model and applicability: Explain why the chapter uses GPT-Neo as its reference model and why its topics are not limited to GPT-Neo.
Why it matters: The reader can treat GPT-Neo as an example rather than a requirement for applying the chapter’s ideas.
Key terms: GPT-Neo, GPT-3 architecture, Hugging Face Transformers, Accelerate, domain-specific models
4.1.1 Text completion
This section shows how to load GPT-Neo for text completion, check its memory and device placement, and generate completions for one or more prompts.
- GPT-Neo loading and placement: Explain how the section loads GPT-Neo and its tokenizer, and what automatic device placement does when the model does not fit entirely on the GPU.
Why it matters: Automatic placement lets you run inference even when available GPU memory cannot hold the entire model. - Model memory and device checks: Describe what Accelerate’s memory estimate tells you and how hf_device_map helps check where the loaded model resides.
Why it matters: A loading-size estimate alone does not establish the total memory required to run inference. - Prompt completion workflow: Describe how the section generates readable text completions for a prompt and how the same task can be performed for a batch of prompts.
Why it matters: Generating and decoding are the steps that turn prompts into readable completions.
Key terms: GPTNeoForCausalLM, GPT2Tokenizer, device_map, auto, Accelerate, offloads, accelerate estimate-memory, Total Size, float32, hf_device_map
4.1.2 Few-shot learning
This section introduces few-shot learning as using examples in a prompt to guide a GPT-based model on new inputs.
- Few-shot prompting with examples: Explain that a GPT-based model can use a few examples in a prompt to generalize to unseen examples, such as in text classification. Describe that the generated output includes the input prompt.
Why it matters: A few examples can show the model the kind of response you want without limiting it to those examples.
Key terms: few-shot learning, prompt, few-shot text classification
4.1.3 Code generation
This section shows that a language model can generate code and explains why its output still needs review and model customization.
- Code Generation Output: Explain that the model can generate code, not just natural language, and describe what the example output contains and why it is truncated.
Why it matters: It helps you interpret what a code-generation request actually returns. - Generated Code Reliability: Explain why generated code requires developer review and why an uncustomized pretrained model may produce poor results.
Why it matters: It keeps a demonstration of code generation from being mistaken for a guarantee of usable code.
Key terms: code generation, Pile dataset, max_length, developer review, lint, fine-tuning, LoRA, RAG, hallucinations
4.1.4 Evaluating the generated content
This section explains why to evaluate generated content before deployment and how to use LM-Eval to run a task-based evaluation.
- Predeployment output quality checks: Explain why an LLM’s output should be evaluated before production deployment and what that evaluation helps decide.
Why it matters: An evaluation result is useful when it informs a deployment decision. - LM-Eval task coverage: Describe what LM-Eval provides for evaluating different models and tasks, including how its flexibility and task versioning help.
Why it matters: A shared framework makes it easier to evaluate models against suitable tasks and reproduce the tests. - Hugging Face evaluation setup: Describe how the section sets up LM-Eval and which options identify a Hugging Face model, task, and device for an evaluation run.
Why it matters: These settings tell LM-Eval exactly which model to test, on what task, and where to run it. - Wikitext evaluation metrics: Describe what the wikitext example evaluates and what its three reported metrics measure.
Why it matters: The example shows that an evaluation task determines the data and metrics used to assess a model.
Key terms: LLM evaluation, output quality, Evaluation Harness, LM-Eval, custom tasks, task versioning, main.py, model, model_args, pretrained
4.2 Calculating inference cost
This section shows how to estimate GPU memory for LLM inference and identifies serving-performance measures beyond model size.
- Inference memory estimate: Explain how parameter count, numerical precision, and overhead affect the GPU memory needed to load a model for inference.
Why it matters: A memory estimate helps you choose hardware that can hold the model during inference. - Lower precision for deployment: Explain why loading a pretrained model in FP16 can help it fit within GPU memory constraints.
Why it matters: Precision is a deployment choice that can change whether a model fits on available hardware. - Serving performance measures: Distinguish GPU utilization, time to first token, time per output token, latency, and throughput when assessing LLM serving.
Why it matters: A model that fits in memory may still respond too slowly or serve too few users.
Key terms: trainable parameters, in-memory size, precision, FP32, FP16, INT8, mixed precision, hardware constraints, GPU power utilization, time to first token
4.3 Areas for improvement (cost savings and performance)
This introduction connects inference-serving metrics to finding cost and performance improvements before the chapter examines specific approaches.
- Metrics for inference improvements: Explain how inference-serving metrics help identify opportunities to reduce costs while making effective use of computational power and keeping users engaged.
Why it matters: Performance measurements help you decide where inference serving needs improvement rather than optimizing blindly.
Key terms: metrics, inference serving, computational power
4.3.1 Getting the most from your GPU
This section uses GPU specifications and attention arithmetic intensity to diagnose GPT-Neo inference performance, and explains why KV caching speeds up token generation.
- GPU capacity and performance limits: Explain how compute throughput, memory capacity, and memory bandwidth affect the choice and use of a GPU for inference. Describe how the T4's compute throughput and bandwidth yield an ops/byte threshold of 216.6.
Why it matters: These limits help you tell whether inference is constrained by memory capacity, data movement, or computation. - KV cache during generation: Trace the prefill and autoregressive sampling phases and explain why retaining the key–value cache speeds up generation despite using additional memory.
Why it matters: Caching trades some available memory for substantially faster next-token generation. - Attention arithmetic intensity diagnosis: Explain arithmetic intensity as computation divided by bytes accessed, and compare it with the GPU's ops/byte ratio to diagnose the GPT-Neo attention workload on a T4.
Why it matters: The comparison identifies data transfer, not a shortage of compute units, as the apparent bottleneck in this example.
Key terms: compute throughput, memory, memory bandwidth, ops/byte ratio, prefill, autoregressive sampling, key–value (KV) cache, use_cache, arithmetic intensity, memory bound
4.3.2 Batching
This section explains how batching improves LLM inference throughput, how GPU memory limits batch size, and when the added latency makes batching unsuitable.
- Static and dynamic batching: Distinguish static batching from dynamic batching and explain how running prompts together can improve GPU utilization.
Why it matters: Batching can produce more responses from the same GPU work when requests can be processed together. - GPU memory batch limit: Explain how available GPU memory and KV-cache use constrain the number of sequences that fit in a batch, using the GPT-Neo example.
Why it matters: A batch that exceeds available memory cannot deliver the expected throughput benefit. - Throughput and latency tradeoff: Describe when batching is useful and why it may be a poor choice for latency-sensitive applications.
Why it matters: The best inference setup depends on whether processing more work overall or responding immediately matters most.
Key terms: static batching, dynamic batching, arithmetic intensity, memory-bound, GPU memory, VRAM, KV cache, prefill phase, sequence length, throughput
4.3.3 Estimating the generation time
This section estimates LLM generation time from GPU memory bandwidth and prefill time, then identifies limits of the estimate and possible inference improvements.
- Per-token bandwidth estimate: Explain how to approximate single-token generation time from model size, parameter precision, and GPU memory bandwidth, using the GPT-Neo example.
Why it matters: This gives you a quick generation-speed estimate before running the model. - Prefill and total generation time: Describe how prefill changes the total-time estimate and state why the estimate may differ from observed time.
Why it matters: Including prefill and recognizing overhead makes a rough speed prediction more useful. - Inference improvement options: Distinguish the inference techniques listed in the section by their stated effects or approaches.
Why it matters: These options point to different ways to improve inference beyond estimating its runtime.
Key terms: trainable parameters, bytes per parameter, memory bandwidth, full precision, prefill phase, forward pass, generation time, overhead, operator fusion, quantization
4.3.4 Optimizing GPU use with DeepSpeed
This section shows how to run GPT-Neo inference with DeepSpeed and evaluate its latency benefit against a possible accuracy tradeoff.
- DeepSpeed inference optimizations: Explain how DeepSpeed can reduce inference memory use and latency through model partitioning and optimized kernels, without requiring model export or new checkpoints for compatible models.
Why it matters: These optimizations can make inference faster or help a model fit within available GPU memory. - GPT-Neo inference setup: Describe the section’s GPT-Neo setup: initialize DeepSpeed and CUDA, load the model in half precision, and configure inference with automatic layer replacement and kernel injection. Distinguish the notebook-specific launcher workaround from steps needed outside notebooks.
Why it matters: The setup shows which choices make the optimized model runnable and comparable on the example’s single GPU. - Latency and accuracy comparison: Describe how the section compares vanilla and DeepSpeed-optimized GPT-Neo and explain why the measured latency gain must be weighed against output quality.
Why it matters: A faster model is only a useful improvement if its output still meets the application’s quality requirements.
Key terms: DeepSpeed, model parallelism, inference-specific kernels, kernel injection, GPT-Neo, half precision, CUDA, replace_method, replace_with_kernel_inject, latency
4.4 Summary
Where this is used later
- 4-bit quantization with GPTQ (chapter 6) · The use of a model to generate content during inference.
- From natural language to protein structures (chapter 8) · Generating a continuation with a language model
- From CIF files to crystal structures (chapter 8) · Autoregressive text completion
- FastAPI (chapter 11) · Text generation from an input prompt.
- Generating Python code with a Transformer architecture (chapter 7) · Applies code generation with a Transformer architecture.
- Using Transformers to generate code (chapter 7) · The earlier discussion of code generation.
- Python code generation with better models (chapter 7) · Assumes familiarity with generating code from prompts.
- Model evaluation (chapter 7) · Motivates evaluation of code-generating models.
Concepts explained in this chapter
- prefillThe inference phase in which prompt tokens are processed in parallel and the KV cache is populated before generation begins.
- arithmetic intensityThe number of compute operations divided by the number of bytes moved to and from memory for an algorithm or model layer.
- quantizationMethods that reduce numeric precision or representational size of model values to improve efficiency, especially for inference; here the absmax method is a specific quantization procedure.
- generationThe task of producing output sequences from a model given an input context or prompt, here applied to proteins rather than natural language.
- DeepSpeedAn open-source deep learning optimization library that supports distributed training and inference and is used here to speed up and memory-optimize Transformer models.
- attentionA sequence-modeling technique that weights relevant parts of an input more heavily than less relevant parts.
- compute-boundA performance regime where execution is limited by available compute units rather than memory transfer rate.
- hardware accelerationThe use of specialized hardware such as GPUs to speed up computation by offloading work from the CPU.
- inference samplingThe process of generating tokens from a Transformer during inference, including prefill and autoregressive decoding.
- ops/byte ratioThe number of floating-point operations available per byte of memory accessed on a given device.
- time per output tokenThe time required to generate each additional output token during response generation.
- time to first tokenThe delay between a request being submitted and the model producing its first output token.
- language modelA very large neural language model trained on broad text corpora to predict or generate language, used here for code generation and assistance.
- mixed precisionUsing reduced-precision numeric formats for some computations or weights to reduce memory use, often with limited loss in performance.
- perplexityA measure of how well a language model predicts tokens in sequence, with lower values generally indicating better predictive fit.
- transformersThe Hugging Face library for working with pretrained transformer models, tokenizers, pipelines, and related training and inference utilities.
- tokenizerA model-specific component that converts concatenated instruction/output text into token sequences for fine-tuning.
- kernel fusionAn optimization that combines adjacent operators into one to reduce overhead and improve inference latency.
- transformer architectureA neural network architecture for sequence modeling built around self-attention and parallel processing, underlying both large and small language models.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.