Prefill
Prefill is the initial inference phase in which a model processes the available input context and prepares state for subsequent autoregressive generation.
Quantization and Fast Inference
Prefill is the initial computation that processes a language-model prompt before token generation begins.
- How it is used
- Frames prefill as a latency-to-first-token bottleneck when choosing CPU or GPU serving for interactive workloads.
- How deep it goes
- Defines prefill as a one-shot prompt pass and identifies it as compute-bound and parallelizable, contrasting it with decode.
- Examples
- Compares 4,000-token prompt latency on an L4 and c3 CPU, citing RAG context, code-file context, and conversation history.
- What you can do after
- Evaluate whether CPU serving can meet first-token latency needs for workloads with long prompts.
Domain-Specific Small Language Models
The inference phase in which prompt tokens are processed in parallel and the KV cache is populated before generation begins.
- How it is used
- Frames prompt processing as the first phase of inference and accounts for its contribution to total generation latency.
- How deep it goes
- Defines parallel prompt processing, explains KV-cache use, and estimates prefill time without modeling additional inference overhead in detail.
- Examples
- Uses a 300-token GPT-Neo prompt on a T4 GPU to estimate prefill time at about 50 ms.
- What you can do after
- Evaluate inference latency by adding estimated prefill time to estimated token-generation time.
Read how Domain-Specific Small Language Models explains prefill →
CUDA for LLMs
Prefill is the inference phase that processes an input prompt and prepares its attention cache for generation.
- How it is used
- Processes an entire prompt in parallel during transformer inference, before sequential token generation.
- How deep it goes
- Distinguishes parallel prompt processing from sequential decode by workload type, but gives no prefill-specific kernel walkthrough.
- What you can do after
- Understand why prompt processing and token generation stress different GPU paths: prefill is GEMM-dominated, while decode is sequential and GEMV-dominated.
Also covered in
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
What to know first
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- forward passA forward pass is a computation that propagates inputs through a model or dependency structure in the forward direction to calculate outputs or intermediate values.
- memory usageMemory usage is the amount of computer memory occupied, allocated, or required by a program, system, data structure, or computation during operation.
Related concepts
- decode phasecontrasts with · The decode phase is the autoregressive inference stage in which a model generates output tokens sequentially from previously processed context.
- key-value cacheuses · A key-value cache stores attention keys and values computed for earlier tokens so they can be reused during subsequent autoregressive generation.
- matrix-matrix multiplicationuses · Matrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- inferencepart of · Inference is the process of applying a trained model to input data to produce predictions or other outputs.
- autoregressive generationpart of · Autoregressive generation produces a sequence one element at a time, conditioning each new element on the elements already available.
Questions about prefill
What is prefill?
Prefill is the initial inference phase in which a model processes the available input context and prepares state for subsequent autoregressive generation.
Which Manning books explain prefill?
Quantization and Fast Inference, Domain-Specific Small Language Models and CUDA for LLMs.
What should I know before learning prefill?
Manning books that teach prefill usually introduce attention, transformer architecture, forward pass and memory usage first.
What is the difference between prefill and decode phase?
Prefill is the initial inference phase in which a model processes the available input context and prepares state for subsequent autoregressive generation. The decode phase is the autoregressive inference stage in which a model generates output tokens sequentially from previously processed context.
What is prefill part of?
Prefill is part of inference. Inference is the process of applying a trained model to input data to produce predictions or other outputs.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


