Latency is the elapsed time between a user input or system request and the corresponding model response or result. It shapes whether an AI feature feels responsive, especially when people must collaborate with it rather than wait passively. section 5.4.1 section 9.3.1
Treat latency as a product requirement rather than an isolated model metric. Users may accept a longer wait for a complete draft but expect prompt responses when making edits. The right threshold therefore depends on the task, intervention model, and surrounding experience. section 5.4.1 section 9.3.1
Evaluate latency under realistic conditions:
- Establish a baseline with the selected model and real-world task and dataset. section 5.4.1
- Test production-like data volumes, API calls, database queries, and real-time streams rather than relying on sandbox performance. section 12.1.3
- Compare response speed with quality, deployment cost, infrastructure demands, and the amount of human involvement the workflow requires. section 9.3.1 section 5.3.2
- Discuss bottlenecks and optimization trade-offs with engineers instead of treating slow responses as an isolated team failure. section 12.1.3
Sandbox speed versus production latency
A model generates predictions in under two seconds in a sandbox. Live traffic may still slow the result because larger data volumes, API calls, and database queries add delay. A load test can expose these bottlenecks before release. section 12.1.3
