Latency is the elapsed time between a request and its response. For a deployed model, it matters alongside model performance: a model that works during training may respond differently after optimization or deployment to an edge device. Test the final, optimized version in its target environment. section 2.1.5
Make latency observable in production:
- Track API response time alongside failing requests and resource utilization to spot slow responses and investigate performance expectations. section 11.1.1
- For an LLM application, capture latency with traces of retrieval and generation, then inspect it alongside prompts, responses, and token usage. section 12.4.3
Latency measured on a training GPU need not predict behavior on an embedded device. Serving at high request volume also presents different challenges from evaluation workloads. Measure the deployed system rather than assuming notebook-scale results will hold. section 2.1.5 section 2.2
