chapter four

4 Engineering system performance evaluations

 

This chapter covers

  • Introducing engineering system performance evaluations
  • Understanding why latency, load time, reliability, and cost matter
  • Evaluating AI systems with shadow traffic before user exposure
  • Designing latency degradation experiments to measure product sensitivity
  • Tracking key system performance metrics for AI architectures
  • Understanding the limitations of engineering performance evaluations

What lies behind every AI model? Data? Definitely. Offline evaluations? Of course. An A/B test? I sure hope so. But there is something else that is just as important: the engineering system that fetches data, invokes the model, serves the output, handles failures, and turns a prediction into an actual product experience.

A model can look excellent in a notebook and still fail in production. It might be too slow, too expensive to serve, too fragile under peak traffic, or too dependent on upstream services that are unreliable. It might improve offline metrics while making the product feel noticeably worse to users. This is why engineering system performance evaluations deserve their own chapter.

4.1 Why latency and load time really matter

4.1.1 Illustrating the impact on user experience and business metrics

4.1.2 Technical constraints and scaling challenges

4.2 Engineering system performance metrics

4.2.1 Key load time metrics

4.2.2 Key latency metrics

4.2.3 LLM and agent-specific performance metrics

4.2.4 Combining load time and latency metrics for performance evaluations

4.3 Shadow traffic

4.3.1 How shadow traffic works

4.3.2 Shadow traffic versus A/B testing

4.3.3 Benefits and challenges of shadow traffic

4.3.4 Mimicking shadow traffic

4.4 Latency degradation experiments

4.4.1 4.4.1 Designing latency degradation tests

4.4.2 Key online evaluation system performance metrics

4.4.3 Movie recommendations example

4.5 Engineering considerations

4.5.1 4.5.1 Make sure to capacity plan your infrastructure resources

4.5.2 4.5.2 You’ll never regret time spent testing under realistic product conditions

4.5.3 4.5.3 Keep in mind these performance optimization tactics

4.6 Summary