liveBook · Manning

Time to first token

As explained in AI Model Evaluation

Time to first token is the elapsed time from a request until the first generated token becomes available to the user.

Time to first token (TTFT) measures the elapsed time from a request until the first generated token becomes visible. In chat and streaming interfaces, it often shapes perceived speed more strongly than total response time: a blank screen feels slow, while early streaming signals progress. TTFT therefore connects system performance to user experience and product outcomes, not just model quality. section 4.1.1 section 4.2.3

Evaluate TTFT alongside the rest of the response lifecycle:

  • Separate prompt processing and retrieval latency from generation latency to locate the delay before output begins. section 4.2.3
  • Pair TTFT with tokens per second and time to final token so early responsiveness does not hide slow completion. section 4.2.3
  • Track context size, cost per request, timeout rate, retry rate, and fallback rate because responsiveness is only one production constraint. section 4.2.3 section 4.6
  • Measure where users experience delay rather than assuming model inference latency equals page latency. section 4.6

Streaming chatbot response

A chatbot that returns a complete answer after six seconds may feel slow if the screen remains blank. Streaming the first token quickly can make the same system feel faster while generation continues, making TTFT more relevant to the experience than total generation time. section 4.1.1

How AI Model Evaluation covers it

How deep it goes
Defines TTFT and situates it among prompt, retrieval, generation, completion, cost, and failure metrics without prescribing an implementation.
Examples
Uses a six-second chatbot response and early token streaming to contrast blank-screen delay with visible progress.
What it assumes
Assumes familiarity with generated tokens, streaming interfaces, and basic latency measurement.
What you can do after
Evaluate LLM experiences by measuring first output, continued generation, completion time, and surrounding production bottlenecks.
How it connects
Connects TTFT to perceived latency, streaming, tokens per second, time to final token, retrieval latency, context size, and cost per request.

Also explained in

Compare how 6 books explain time to first token →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.