liveBook · Manning

Prefill

As explained in Quantization and Fast Inference

Prefill is the initial computation that processes a language-model prompt before token generation begins.

Prefill processes the prompt in a one-shot pass before token generation. It is compute-bound and parallelizable, unlike decode, so it strongly affects latency to the first token. A 4,000-token prompt takes an L4 tens of milliseconds; a c3 CPU instance takes seconds. For interactive workloads with long prompts—such as RAG context, code-file context, or growing conversation history—that delay can rule out CPU serving. section 10.1 section 10.1

How Quantization and Fast Inference covers it

How deep it goes
Defines prefill as a one-shot prompt pass and identifies it as compute-bound and parallelizable, contrasting it with decode.
Examples
Compares 4,000-token prompt latency on an L4 and c3 CPU, citing RAG context, code-file context, and conversation history.
What you can do after
Evaluate whether CPU serving can meet first-token latency needs for workloads with long prompts.
How it connects
Connects prefill to decode and latency-to-first-token decisions.

Also explained in

Compare how 3 books explain prefill →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.