Time to first token (TTFT) measures the elapsed time from a user query until a generative model returns its first output token. Lower TTFT generally creates a more responsive streaming experience. It includes prompt processing and generation of the first token, so it differs from time per output token and full end-to-end latency. section 11.4.1 section 11.4.2
Interpret TTFT alongside the factors and metrics that shape production behavior:
- Account for prompt size because processing the input contributes to first-token latency; retrieval-augmented generation can make prompts substantially larger. section 11.4.2
- Separate TTFT from time per output token (TPOT), which measures ongoing token-generation speed but does not account for the input prompt. section 11.4.1 section 11.4.2
- Compare similar workloads rather than relying on end-to-end latency alone, because generated token counts can vary greatly even for similarly sized prompts. section 11.4.2
- Balance model capability against responsiveness: larger models are more capable but generally consume more resources and take longer. section 11.4.2
- Track TTFT with TPOT, latency, and throughput, using observability tools such as MLflow where appropriate. section 11.7
Measuring first-token responsiveness
A latency test can call the Azure OpenAI Chat API through REST, record the start time, send a payload with streaming enabled, and issue several requests concurrently with ThreadPoolExecutor. Use comparable prompts and request sizes when comparing TTFT results. section 11.4.2
Table 11.3 LLM inference metrics
|
Metric
|
Definition
|
|---|---|
|
Time to first token (TTFT)
|
Measures the time it takes for the model to generate the first token after a user query. Lower TTFT means a more responsive user experience. TTFT is influenced by the time required to process the prompt and generate the first output token.
|
|
Time-per-output token (TPOT)
|
Calculates the time required for the model to generate one token for a specific query. Lower TPOT means faster text generation. The model size, the hardware configuration, and the decoding algorithm influence TPOT.
|
|
Latency
|
This metric measures the time it takes for data to move from its starting point to its destination. In the case of LLMs, it is the time for the model to generate a response to the user. The model and the tokens generated influence LLMs’ latency. Generally, most of the time is spent generating complete tokens, which are generated one at a time. The longer the generation, the higher the latency.
|
|
Throughput
|
Measures the amount of data that can be transferred in a unit of time. In this case, the number of output tokens per second on a deployment unit can be served across all requests.
|
|
Request per second (RPS)
|
RPS measures the throughput of LLMs in production and indicates the number of requests an LLM can handle every second. This metric is crucial for understanding the scalability and efficiency of LLMs when deployed in real-world applications.
|
Compare TTFT with TPOT, latency, and throughput in a compact reference table. section 11.4.1
