Time to first token (TTFT) is the elapsed time from a generation request until the model produces its first output token. It measures a user's wait for a response, not the time to complete a generation or the rate of subsequent token production. section 4.3.5
TTFT matters in interactive applications and latency-sensitive agent loops. When an agent call blocks its next reasoning step, low-batch TTFT and per-token decode speed matter more than peak throughput at large batch sizes; batching that improves throughput can make individual calls wait. section 4.3.5 section 12.8
Table 4.4 Inference performance benchmarks
|
Performance
|
Inference Time
|
TTFT
|
Throughput
|
|
Base model
|
2.834s
± 0.041s
|
64.49ms
± 1.0ms
|
17.65 tokens/s
± 0.26
|
|
Data guided
|
2.429s
± 0.009s
(+14.3%)
|
56.30ms
± 4.05ms
(+12.7%)
|
20.59 tokens/s
± 0.08
(+16.7%)
|
|
Data guided protected
|
2.426s
± 0.010s (+14.4%)
|
54.40ms
± 1.78ms (+15.6%)
|
20.61 tokens/s
± 0.09 (+16.8%)
|
Compare reported TTFT for the base model and two depth-pruned variants. Measurements used a T4 GPU in Google Colab, with GPU warmup and 10 records per dataset; results can vary between runs. section 4.3.5





