liveBook · Manning

Latency

As explained in AI Model Evaluation

The total elapsed time from receiving a user request to displaying the complete response, including all intermediate processing steps.

Latency is the elapsed time from a user request to the display of the complete response. End-to-end latency includes request handling, feature gathering, model inference, response formatting, and rendering. It matters because a model can produce accurate predictions yet still be unusable if the surrounding system responds too slowly. section 4.2.2 section 2.1.1

Inference latency measures only the time the model takes to produce a prediction. A slow database query, network transfer, or rendering step can also delay the response, so fast inference does not guarantee a fast user experience. First establish an end-to-end baseline, then measure the individual stages to find the actual bottleneck. section 4.4.2 section 6.4.1

Measure latency from the perspectives that matter to the product:

  • Real latency measures how long the system takes; perceived latency describes how slow it feels. Streaming or progressive rendering can improve the experience even when backend work continues. section 4.1.1
  • For streaming LLM responses, track time to first token, tokens per second, and time to final token. Measure retrieval separately when it is part of the request path. section 4.2.3
  • For multi-step agents, track tool latency and total task completion time alongside the number of model and tool calls. section 4.6

Evaluate latency alongside model quality, throughput, reliability, and cost. Quantization, pruning, distillation, or different hardware may reduce inference time. Batching can improve throughput and hardware utilization but may worsen responsiveness for individual users; reducing model complexity may also reduce prediction quality. section 4.2.2 section 4.2.4

Testing movie recommendation delays

A latency degradation A/B test adds delays near the recommendation page’s frontend and compares a control with five slower variants. Track page-load percentiles alongside click-through, bounce, abandonment, and downstream viewing, and replicate by device type. Because the test deliberately worsens the experience, limit exposure and define guardrails and stop conditions. section 4.4.3 section 4.6

Table 4.4 Latency Degradation Test Results on Mobile
Variant Cell
Added Latency
Avg Page Load Time
CTR
Bounce Rate
Control
0ms
1050ms
23.5%
27.8%
Test Variant 1
100ms
1155ms
22.7%
29.1%
Test Variant 2
200ms
1250ms
20.2%
32.7%
Test Variant 3
300ms
1360ms
16.4%
39.5%
Test Variant 4
400ms
1455ms
12.2%
48.3%
Test Variant 5
500ms
1552ms
8.7%
56.9%

The mobile results illustrate how added latency can be compared with page load time, click-through rate, and bounce rate across variants. section 4.4.3

Use latency as a real-time A/B test guardrail, segmented by model version, treatment group, geography, and device. A slower treatment can make a model appear less effective because the experience is slower rather than because its predictions are less relevant. Shadow traffic can compare candidate-system latency without exposing users to its output, but it cannot establish user preference. section 6.4.1 section 4.6

How AI Model Evaluation covers it

How deep it goes
Defines end-to-end and inference latency, then outlines stage-level measurement, optimization trade-offs, and online experiments.
Examples
Uses a movie recommendation latency degradation test and a simulated shadow-traffic comparison.
What you can do after
Evaluate latency bottlenecks and trade-offs, and design experiments that relate delays to user behavior.
How it connects
Connects latency to throughput, model quality, perceived performance, A/B testing, and shadow traffic.

Also explained in

Compare how 12 books explain latency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.