Time per output token (TPOT) measures how long an inference system takes to generate each output token for each active user. It complements time to first token (TTFT), latency, and throughput when assessing production serving performance. section 4.2
Use TPOT to reason about generation speed and deployment constraints:
- Distinguish TPOT from TTFT: TPOT describes ongoing token generation, while TTFT describes the delay before the first token appears. section 4.2
- Interpret complete-response latency through both TTFT and TPOT; evaluate throughput separately across users and requests. section 4.2
- Account for GPU utilization, memory constraints, KV-cache capacity, batching, and optimization when targeting TPOT and other serving objectives. section 4.2 section 4.4
GPT-Neo on a T4 GPU
A GPT-Neo inference estimate on a T4 GPU produces about one token every 36 milliseconds, illustrating TPOT as an observable generation-time measure. section 4.4
