A service can finish individual requests quickly yet struggle when many users arrive at once. Throughput measures how much work it completes over time. For an LLM service, queries per second (QPS) measures completed requests; tokens per second (TPS) measures how fast a response stream delivers output. Both matter, but they answer different questions. section 6.3.2
Streaming makes TPS especially useful because generated responses vary greatly in length. Users can start reading before generation finishes, so total request latency alone misses part of their experience. Time to first token (TTFT) captures when feedback begins; TPS captures the pace afterward. About 11 TPS for English is offered as a practical target—slightly faster than most people read—not a guarantee that every response will feel fast. section 6.1.5 section 6.3.2
Use throughput alongside latency and resource measurements when making service decisions:
- Try adaptive request batching when traffic and latency budget allow it. Larger batch sizes or longer batch delays can improve throughput and resource use, but make requests wait longer; very sparse traffic or strict latency needs may justify skipping batching. section 6.1.3
- Load-test before production or an expected traffic peak. Increase requests until the service fails or autoscales, then inspect throughput and latency at that point to estimate capacity and replica needs. section 6.3.2
- Watch CPU, GPU, and memory usage as load rises. A throughput number without those measurements can hide an overworked service or an autoscaling problem. section 6.3.2
A small-device reality check
An LLM served on a Raspberry Pi may reach only about five tokens per second at best. A working chat interface therefore proves that the deployment runs, not that its output pace suits users; the prototype can instead make the case for more capable hardware. section 11.3
Listing 6.12 Load testing with Locust
import time
from locust import HttpUser, task, events
stat_file = open("stats.csv", "w") #1
stat_file.write("Latency,TTFT,TPS\n")
class StreamUser(HttpUser):
@task
def generate(self):
token_count = 0 #2
start = time.time()
with self.client.post( #3
"/generate",
data='{"prompt": "Salt Lake City is a"}',
catch_response=True,
stream=True,
) as response:
first_response = time.time()
for line in response.iter_lines(decode_unicode=True):
token_count += 1
end = time.time() #4
latency = end - start
ttft = first_response - start
tps = token_count / (end - first_response)
stat_file.write(f"{latency},{ttft},{tps}\n") #5
# Close stats file when Locust quits
@events.quitting.add_listener
def close_stats_file(environment):
stat_file.close()
The Locust example records latency, TTFT, and a calculated TPS while consuming a streamed response. Check its measurement boundaries before treating the results as exact token metrics: it timestamps the first response before reading the stream and counts response lines. section 6.3.2
