Latency is the elapsed time from a user request to the display of the complete response. End-to-end latency includes request handling, feature gathering, model inference, response formatting, and rendering. It matters because a model can produce accurate predictions yet still be unusable if the surrounding system responds too slowly. section 4.2.2 section 2.1.1
Inference latency measures only the time the model takes to produce a prediction. A slow database query, network transfer, or rendering step can also delay the response, so fast inference does not guarantee a fast user experience. First establish an end-to-end baseline, then measure the individual stages to find the actual bottleneck. section 4.4.2 section 6.4.1
Measure latency from the perspectives that matter to the product:
- Real latency measures how long the system takes; perceived latency describes how slow it feels. Streaming or progressive rendering can improve the experience even when backend work continues. section 4.1.1
- For streaming LLM responses, track time to first token, tokens per second, and time to final token. Measure retrieval separately when it is part of the request path. section 4.2.3
- For multi-step agents, track tool latency and total task completion time alongside the number of model and tool calls. section 4.6
Evaluate latency alongside model quality, throughput, reliability, and cost. Quantization, pruning, distillation, or different hardware may reduce inference time. Batching can improve throughput and hardware utilization but may worsen responsiveness for individual users; reducing model complexity may also reduce prediction quality. section 4.2.2 section 4.2.4
Testing movie recommendation delays
A latency degradation A/B test adds delays near the recommendation page’s frontend and compares a control with five slower variants. Track page-load percentiles alongside click-through, bounce, abandonment, and downstream viewing, and replicate by device type. Because the test deliberately worsens the experience, limit exposure and define guardrails and stop conditions. section 4.4.3 section 4.6
Table 4.4 Latency Degradation Test Results on Mobile
|
Variant Cell
|
Added Latency
|
Avg Page Load Time
|
CTR
|
Bounce Rate
|
|
Control
|
0ms
|
1050ms
|
23.5%
|
27.8%
|
|
Test Variant 1
|
100ms
|
1155ms
|
22.7%
|
29.1%
|
|
Test Variant 2
|
200ms
|
1250ms
|
20.2%
|
32.7%
|
|
Test Variant 3
|
300ms
|
1360ms
|
16.4%
|
39.5%
|
|
Test Variant 4
|
400ms
|
1455ms
|
12.2%
|
48.3%
|
|
Test Variant 5
|
500ms
|
1552ms
|
8.7%
|
56.9%
|
The mobile results illustrate how added latency can be compared with page load time, click-through rate, and bounce rate across variants. section 4.4.3
Use latency as a real-time A/B test guardrail, segmented by model version, treatment group, geography, and device. A slower treatment can make a model appear less effective because the experience is slower rather than because its predictions are less relevant. Shadow traffic can compare candidate-system latency without exposing users to its output, but it cannot establish user preference. section 6.4.1 section 4.6











