A streaming service may complete a request quickly yet feel slow if users wait before seeing any output. Time to first token (TTFT) measures that initial wait: the elapsed time from request submission until the first generated token arrives. section 6.3.2
Use TTFT with complementary service and user-experience metrics:
- Compare TTFT with total latency, which measures complete-request duration. section 6.3.2
- Track tokens per second (TPS) to measure streaming speed after the first token. section 6.3.2
- Record CPU, GPU, and memory utilization during load tests to expose resource pressure. section 6.3.2
Measuring TTFT with Locust
A streaming Locust request records the request start time, captures the first response timestamp, subtracts the two, and writes TTFT alongside latency and TPS to CSV. section 6.3.2
Listing 6.12 Load testing with Locust
import time
from locust import HttpUser, task, events
stat_file = open("stats.csv", "w") #1
stat_file.write("Latency,TTFT,TPS\n")
class StreamUser(HttpUser):
@task
def generate(self):
token_count = 0 #2
start = time.time()
with self.client.post( #3
"/generate",
data='{"prompt": "Salt Lake City is a"}',
catch_response=True,
stream=True,
) as response:
first_response = time.time()
for line in response.iter_lines(decode_unicode=True):
token_count += 1
end = time.time() #4
latency = end - start
ttft = first_response - start
tps = token_count / (end - first_response)
stat_file.write(f"{latency},{ttft},{tps}\n") #5
# Close stats file when Locust quits
@events.quitting.add_listener
def close_stats_file(environment):
stat_file.close()
Inspect the complete Python load-test listing, including streaming iteration, timestamp capture, and CSV reporting. section 6.3.2
