liveBook · Manning

Time to first token

As explained in Domain-Specific Small Language Models

The delay between a request being submitted and the model producing its first output token.

Time to first token (TTFT) measures the delay between submitting a prompt and seeing the model’s first output token. It captures the initial responsiveness of an inference service. section 4.2

TTFT belongs to a broader set of serving metrics:

  • Time per output token (TPOT) measures generation time for each active user. section 4.2
  • Latency covers the complete response and depends on TTFT and TPOT. section 4.2
  • Throughput measures output tokens generated per second across users and requests. section 4.2

Production inference systems use TTFT alongside hardware utilization, throughput, and latency targets. Optimization and quantization can help meet these targets while reducing serving hardware requirements. section 4.2 section 4.4

How Domain-Specific Small Language Models covers it

How deep it goes
Defines TTFT and distinguishes it from TPOT, latency, and throughput without detailing a measurement procedure.
Examples
Uses the interval from a submitted prompt to the model’s first output token as the defining operational example.
What you can do after
Evaluate initial inference responsiveness and relate it to complete-response latency and serving targets.
How it connects
Connects TTFT with TPOT, latency, throughput, GPU utilization, quantization, and inference optimization.
Cover of Domain-Specific Small Language Models
liveBookmanning.com
Read Domain-Specific Small Language Models on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Domain-Specific Small Language Models. The section links point to the parts of the book it draws on.