Prefill

As explained in Domain-Specific Small Language Models

The inference phase in which prompt tokens are processed in parallel and the KV cache is populated before generation begins.

Prefill is the first phase of generative Transformer inference. It processes all prompt tokens in parallel, populates the key–value (KV) cache, and produces no generated tokens. Autoregressive sampling then uses that cached attention state to decode tokens one at a time. section 4.3.1

Operationally, prefill matters because it:

• Processes the prompt in one forward pass instead of repeatedly recomputing attention over already-seen prompt tokens. section 4.3.1
• Consumes memory for cached keys and values in GPU VRAM or CPU RAM, so sequence length and batch size are capacity constraints. section 4.3.1 section 4.3.2
• Contributes to end-to-end latency alongside per-token decoding; a practical estimate adds prefill time to the time for generated tokens. section 4.3.3
• Supports higher-throughput dynamic batching when queued requests share a forward pass, but queuing can harm chatbot response latency. section 4.3.2

Estimating a GPT-Neo run

For a 300-token prompt on a T4, an example estimates roughly 50 ms for prefill and 36 ms per generated token. The resulting estimate is 50 + (300 × 36) = 10.85 seconds, while measured execution also includes other overheads. section 4.3.3

Figure 4.1 The (Q × K^T) × V computation process with caching

Use the attention-flow diagram to see how cached keys and values avoid recomputing the full prior sequence during decoding. section 4.3.1

How Domain-Specific Small Language Models covers it

How deep it goes

Explains parallel processing, KV-cache storage, batching, and timing estimates without deriving the attention kernels fully.

Examples

Uses GPT-Neo generation on a T4, KV-cache timing comparisons, dynamic batching, and a 300-token latency estimate.

What it assumes

Assumes familiarity with Transformer inference, attention, GPU memory, and token generation while defining prefill and KV caching operationally.

What you can do after

Evaluate inference latency and memory capacity by estimating prefill cost, KV-cache usage, and batch-size tradeoffs.

How it connects

Connects prefill to autoregressive sampling, KV cache, attention, dynamic batching, arithmetic intensity, and GPU memory.

liveBookmanning.com

Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

• key-value cacheuses · A key-value cache is a data structure that stores previously computed transformer attention keys and values for reuse during subsequent token generation.
• decodecontrasts with · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
• forward passuses · A forward pass is a calculation that propagates inputs through an ordered computational or dependency structure to produce outputs or derived values.
• cachinguses · Caching is the practice of storing reusable data or computation results so later access can avoid repeating retrieval or computation.
• time to first tokenused by · Time to first token is the elapsed time between submitting a language-model generation request and receiving its first generated output token.

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.