liveBook · Manning

Prefill

Prefill is the initial inference phase in which a model processes the available input context and prepares state for subsequent autoregressive generation.

How 3 Manning books explain it

Quantization and Fast Inference

Prefill is the initial computation that processes a language-model prompt before token generation begins.

How it is used
Frames prefill as a latency-to-first-token bottleneck when choosing CPU or GPU serving for interactive workloads.
How deep it goes
Defines prefill as a one-shot prompt pass and identifies it as compute-bound and parallelizable, contrasting it with decode.
Examples
Compares 4,000-token prompt latency on an L4 and c3 CPU, citing RAG context, code-file context, and conversation history.
What you can do after
Evaluate whether CPU serving can meet first-token latency needs for workloads with long prompts.

Read how Quantization and Fast Inference explains prefill →

Domain-Specific Small Language Models

The inference phase in which prompt tokens are processed in parallel and the KV cache is populated before generation begins.

How it is used
Frames prompt processing as the first phase of inference and accounts for its contribution to total generation latency.
How deep it goes
Defines parallel prompt processing, explains KV-cache use, and estimates prefill time without modeling additional inference overhead in detail.
Examples
Uses a 300-token GPT-Neo prompt on a T4 GPU to estimate prefill time at about 50 ms.
What you can do after
Evaluate inference latency by adding estimated prefill time to estimated token-generation time.

Read how Domain-Specific Small Language Models explains prefill →

CUDA for LLMs

Prefill is the inference phase that processes an input prompt and prepares its attention cache for generation.

How it is used
Processes an entire prompt in parallel during transformer inference, before sequential token generation.
How deep it goes
Distinguishes parallel prompt processing from sequential decode by workload type, but gives no prefill-specific kernel walkthrough.
What you can do after
Understand why prompt processing and token generation stress different GPU paths: prefill is GEMM-dominated, while decode is sequential and GEMV-dominated.

Read how CUDA for LLMs explains prefill →

Also covered in

What to know first

Related concepts

Questions about prefill

What is prefill?

Prefill is the initial inference phase in which a model processes the available input context and prepares state for subsequent autoregressive generation.

Which Manning books explain prefill?

Quantization and Fast Inference, Domain-Specific Small Language Models and CUDA for LLMs.

What should I know before learning prefill?

Manning books that teach prefill usually introduce attention, transformer architecture, forward pass and memory usage first.

What is the difference between prefill and decode phase?

Prefill is the initial inference phase in which a model processes the available input context and prepares state for subsequent autoregressive generation. The decode phase is the autoregressive inference stage in which a model generates output tokens sequentially from previously processed context.

What is prefill part of?

Prefill is part of inference. Inference is the process of applying a trained model to input data to produce predictions or other outputs.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.