liveBook · Manning

Time to first token

As explained in Rearchitecting LLMs

Time to first token is the elapsed time from a generation request until the model produces its first output token.

Time to first token (TTFT) is the elapsed time from a generation request until the model produces its first output token. It measures a user's wait for a response, not the time to complete a generation or the rate of subsequent token production. section 4.3.5

TTFT matters in interactive applications and latency-sensitive agent loops. When an agent call blocks its next reasoning step, low-batch TTFT and per-token decode speed matter more than peak throughput at large batch sizes; batching that improves throughput can make individual calls wait. section 4.3.5 section 12.8

Table 4.4 Inference performance benchmarks
Performance
Inference Time
TTFT
Throughput
Base model
2.834s
± 0.041s
64.49ms
± 1.0ms
17.65 tokens/s
± 0.26
Data guided
2.429s
± 0.009s
(+14.3%)
56.30ms
± 4.05ms
(+12.7%)
20.59 tokens/s
± 0.08
(+16.7%)
Data guided protected
2.426s
± 0.010s (+14.4%)
54.40ms
± 1.78ms (+15.6%)
20.61 tokens/s
± 0.09 (+16.8%)

Compare reported TTFT for the base model and two depth-pruned variants. Measurements used a T4 GPU in Google Colab, with GPU warmup and 10 records per dataset; results can vary between runs. section 4.3.5

How Rearchitecting LLMs covers it

How deep it goes
Defines TTFT and reports benchmark comparisons, but does not detail timing instrumentation.
Examples
Compares base and depth-pruned model TTFT on a T4 GPU and describes repeated short agent calls.
What you can do after
Evaluate first-token responsiveness alongside total inference time and throughput when comparing serving configurations.
How it connects
Distinguishes TTFT from inference time and throughput, and relates it to per-token decode speed and batching.

Also explained in

Compare how 6 books explain time to first token →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.