Prefill processes the prompt in a one-shot pass before token generation. It is compute-bound and parallelizable, unlike decode, so it strongly affects latency to the first token. A 4,000-token prompt takes an L4 tens of milliseconds; a c3 CPU instance takes seconds. For interactive workloads with long prompts—such as RAG context, code-file context, or growing conversation history—that delay can rule out CPU serving. section 10.1 section 10.1
Prefill
Prefill is the initial computation that processes a language-model prompt before token generation begins.
How Quantization and Fast Inference covers it
- How deep it goes
- Defines prefill as a one-shot prompt pass and identifies it as compute-bound and parallelizable, contrasting it with decode.
- Examples
- Compares 4,000-token prompt latency on an L4 and c3 CPU, citing RAG context, code-file context, and conversation history.
- What you can do after
- Evaluate whether CPU serving can meet first-token latency needs for workloads with long prompts.
- How it connects
- Connects prefill to decode and latency-to-first-token decisions.
Also explained in
-
Domain-Specific Small Language ModelsThe inference phase in which prompt tokens are processed in parallel and the KV cache is populated before generation begins.
-
CUDA for LLMsPrefill is the inference phase that processes an input prompt and prepares its attention cache for generation.
Compare how 3 books explain prefill →
Related concepts
- decode phasecontrasts with · The decode phase is the autoregressive inference stage in which a model generates output tokens sequentially from previously processed context.
- key-value cacheuses · A key-value cache stores attention keys and values computed for earlier tokens so they can be reused during subsequent autoregressive generation.
- matrix-matrix multiplicationuses · Matrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- inferencepart of · Inference is the process of applying a trained model to input data to produce predictions or other outputs.
- autoregressive generationpart of · Autoregressive generation produces a sequence one element at a time, conditioning each new element on the elements already available.
This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.


