liveBook · Manning

Time to first token

As explained in Quantization and Fast Inference

The elapsed time from submitting a language-model request until its first generated token is available.

Time to first token (TTFT) is the elapsed time from submitting a language-model request until its first generated token is available. It measures the user-visible wait before a response starts; it is distinct from the rate at which subsequent tokens are generated. section 11.1.1 section 11.2.3

Prompt processing, or prefill, drives much of this wait. Prefill processes the prompt, populates the KV cache, and emits the first token. It is compute-bound and parallelizable; decode then extends the response one token at a time and is typically memory-bound. Longer prompts can therefore raise TTFT even when decode speed changes little. section 10.1 section 11.2.3

Prompt length on a phone

For Llama-3.2-1B on a Pixel, TTFT rose from 428 ms for a 32-token prompt to 2.39 s for 1,024 tokens: a 5.6× increase for 32× as many prompt tokens. The increase was sublinear as setup was amortized over the longer sequence. Decode rates stayed within 8% across the sweep, while measured power remained about 5.5 W. section 11.2.3 section 11.1

Set a user-visible latency budget alongside a sustained-power budget, and test representative prompt lengths. For an LLM service-level agreement, use TTFT for the initial response and track decode separately; a comfortable streaming rate does not compensate for a long prefill wait. On edge devices, the summary budget calls for p95 latency on the user-visible path and mean power over a sustained window. section 10.1 section 11.6

Record where inference actually ran, not just the requested backend. In LiteRT-LM 0.12.0 on the Tensor G5, INT8 transformer workloads were routed to the GPU regardless of the CPU or GPU backend hint; the measurements recorded compute_units as gpu. section 11.2.3

How Quantization and Fast Inference covers it

How deep it goes
Explains prefill and decode separately, with a prompt-length sweep and guidance on latency and power budgets.
Examples
Compares Llama-3.2-1B TTFT on a Pixel across 32- and 1,024-token prompts, alongside decode rate and power.
What you can do after
Evaluate language-model configurations against initial-response latency and sustained-power budgets while distinguishing requested from observed compute backends.
How it connects
Connects TTFT to prefill, KV caching, decode throughput, prompt length, and power budgets.

Also explained in

Compare how 6 books explain time to first token →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.