Time to first token (TTFT) is the elapsed time from submitting a language-model request until its first generated token is available. It measures the user-visible wait before a response starts; it is distinct from the rate at which subsequent tokens are generated. section 11.1.1 section 11.2.3
Prompt processing, or prefill, drives much of this wait. Prefill processes the prompt, populates the KV cache, and emits the first token. It is compute-bound and parallelizable; decode then extends the response one token at a time and is typically memory-bound. Longer prompts can therefore raise TTFT even when decode speed changes little. section 10.1 section 11.2.3
Prompt length on a phone
For Llama-3.2-1B on a Pixel, TTFT rose from 428 ms for a 32-token prompt to 2.39 s for 1,024 tokens: a 5.6× increase for 32× as many prompt tokens. The increase was sublinear as setup was amortized over the longer sequence. Decode rates stayed within 8% across the sweep, while measured power remained about 5.5 W. section 11.2.3 section 11.1
Set a user-visible latency budget alongside a sustained-power budget, and test representative prompt lengths. For an LLM service-level agreement, use TTFT for the initial response and track decode separately; a comfortable streaming rate does not compensate for a long prefill wait. On edge devices, the summary budget calls for p95 latency on the user-visible path and mean power over a sustained window. section 10.1 section 11.6
Record where inference actually ran, not just the requested backend. In LiteRT-LM 0.12.0 on the Tensor G5, INT8 transformer workloads were routed to the GPU regardless of the CPU or GPU backend hint; the measurements recorded compute_units as gpu. section 11.2.3





