Time per output token (TPOT) measures how long an LLM takes to generate each output token, typically excluding prompt-processing time. It describes decoding speed after generation begins: tokens are produced sequentially, with each new token informing the next one. section 11.4.1
Use TPOT as one part of an inference-performance picture:
- Lower TPOT means faster text generation, which can improve the experience of applications that stream responses. section 11.4.1
- Interpret TPOT alongside time to first token (TTFT), latency, and throughput rather than treating it as total response time. section 11.4.1 section 11.4.2
- Account for model size, hardware configuration, and decoding algorithm; larger models generally require more time and resources. section 11.4.1 section 11.4.2
- Compare requests with similar prompt and output characteristics, because output length and prompt size can make raw latency comparisons misleading. section 11.4.2
Why output length matters
Generating a short poem about dogs and generating a three-page poem may use similarly sized prompts but require very different numbers of generated tokens. Their end-to-end times therefore should not be compared as equivalent workloads; TPOT helps isolate generation speed, while still excluding prompt-processing cost. section 11.4.2
Table 11.3 LLM inference metrics
|
Metric
|
Definition
|
|---|---|
|
Time to first token (TTFT)
|
Measures the time it takes for the model to generate the first token after a user query. Lower TTFT means a more responsive user experience. TTFT is influenced by the time required to process the prompt and generate the first output token.
|
|
Time-per-output token (TPOT)
|
Calculates the time required for the model to generate one token for a specific query. Lower TPOT means faster text generation. The model size, the hardware configuration, and the decoding algorithm influence TPOT.
|
|
Latency
|
This metric measures the time it takes for data to move from its starting point to its destination. In the case of LLMs, it is the time for the model to generate a response to the user. The model and the tokens generated influence LLMs’ latency. Generally, most of the time is spent generating complete tokens, which are generated one at a time. The longer the generation, the higher the latency.
|
|
Throughput
|
Measures the amount of data that can be transferred in a unit of time. In this case, the number of output tokens per second on a deployment unit can be served across all requests.
|
|
Request per second (RPS)
|
RPS measures the throughput of LLMs in production and indicates the number of requests an LLM can handle every second. This metric is crucial for understanding the scalability and efficiency of LLMs when deployed in real-world applications.
|
Use the inference-metrics table to compare TPOT with TTFT, latency, and throughput and to see which performance questions each metric answers. section 11.4.1
Listing 11.1 Measuring latency
import os
...
AZURE_ENDPOINT = os.getenv("AOAI_ENDPOINT") #1
AOAI_API_KEY = os.getenv("AOAI_KEY") #1
MODEL = "gpt35" #1
API_VERSION = "2024-02-15-preview" #1
headers = {
"api-key": AOAI_API_KEY,
"Content-Type": "application/json"
}
def get_payload(): #2
return {
"model": MODEL,
"max_tokens": 50,
"messages": [{"role": "system", "content":
"You are a helpful assistant."},
{"role": "user", "content": "Hello, world!"}],
"temperature": 0.95,
"stream": True #3
}
def call_api_and_measure_latency(): #4
payload = get_payload()
start_time = time.time() #5
response = requests.post(AZURE_ENDPOINT,
headers=headers,
json=payload, timeout=20)
latency = time.time() - start_time #6
return latency, response.status_code
num_requests = 50 #7
def main():
with ThreadPoolExecutor(max_workers=20) as executor: #8
futures = [executor.submit(call_api_and_measure_latency)
for _ in range(num_requests)]
latencies = []
for future in tqdm(as_completed(futures), total=num_requests):
latency, status_code = future.result()
print(f"Latency: {latency}s, Status Code: {status_code}")
latencies.append(latency)
average_latency = sum(latencies) / len(latencies) #9
print(f"Average Latency: {average_latency}s")
if __name__ == "__main__":
main()
Use the Azure OpenAI REST example to measure round-trip latency under concurrent requests; interpret that result as service latency, not TPOT alone. section 11.4.2
