liveBook · Manning

Chapter 4: Engineering system performance evaluations

What you will learn in chapter 4 of AI Model Evaluation

This introduction explains why evaluating an AI feature requires assessing the production system around the model, not just the model’s offline performance.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction explains why evaluating an AI feature requires assessing the production system around the model, not just the model’s offline performance.

Key terms: engineering system, product experience, engineering system performance evaluations, offline diagnostic evaluations, reliably, quickly, cost-effectively

4.1 Why latency and load time really matter

This section explains why a production AI feature must be served quickly and reliably, and why the user-visible effect of latency matters more than model speed alone.

Key terms: offline quality, under load, scalable engineering architecture, end-to-end latency, user engagement, conversion, backend model inference time, total page load time, model-serving latency, progressive rendering

4.1.1 Illustrating the impact on user experience and business metrics

This section explains how user-visible system delays affect product and business outcomes and why model evaluations must account for responsiveness as well as offline quality.

Key terms: system performance, user engagement, page views, impressions, Ad revenue, real latency, perceived latency, streaming, time to the first token, accuracy

4.1.2 Technical constraints and scaling challenges

Production AI performance can differ sharply from offline results because real traffic, end-to-end request dependencies, and scaling trade-offs create latency that isolated model tests may miss.

Key terms: controlled environment, production traffic, steady-state testing, data distribution shifts, offline and online evaluations, data normalization, feature engineering, text tokenization, data dependencies, middleware overhead

4.2 Engineering system performance metrics

This section introduces the categories of engineering performance metrics and explains why they should be considered together alongside prediction quality.

Key terms: frontend load time metrics, backend latency metrics, model-serving metrics, infrastructure metrics, orchestration metrics

4.2.1 Key load time metrics

This section explains how user-visible load time and model-serving readiness affect AI product performance, and introduces metrics for measuring different parts of that experience.

Key terms: load time metrics, inference latency, CDNs, stack, Time to First Byte, TTFB, First Contentful Paint, FCP, Fully Loaded Time, progressive rendering

4.2.2 Key latency metrics

This section defines two latency metrics for interactive AI systems and explains the factors and serving choices that affect responsiveness.

Key terms: end-to-end latency, Application Performance Monitoring (APM), Model Inference Latency, quantization, pruning, distillation, hardware accelerators, batching, live inference, throughput

4.2.3 LLM and agent-specific performance metrics

This section identifies LLM and agent-specific performance metrics that reveal how quickly and reliably a user receives a response or completes a task.

Key terms: time to first token (TTFT), tokens per second, time to final token, prompt processing time, context size, retrieval latency, cost per request, timeout, retry, and fallback rates, one-shot agent, multi-step agent

4.2.4 Combining load time and latency metrics for performance evaluations

This section shows how to evaluate AI system performance by combining load-time and latency metrics with model quality and product requirements.

Key terms: load time metrics, latency metrics, frontend performance metrics, backend service metrics, AI-specific metrics, infrastructure metrics, model quality, product impact, inference latency, throughput

4.3 Shadow traffic

Introduces shadow traffic as a way to test a candidate system under real production traffic without showing its output to users, and explains what this can and cannot evaluate.

Key terms: shadow traffic, dark loading, production traffic, candidate system, latency, throughput, error rates, resource utilization, fallback behavior, output differences

4.3.1 How shadow traffic works

This section explains how shadow traffic sends copies of real user requests to a new system for evaluation while users continue to receive responses from the production system.

Key terms: shadow traffic, request duplication, traffic routing, production system, shadow system, detailed logging, real production traffic, latency, throughput, resource utilization

4.3.2 Shadow traffic versus A/B testing

This section contrasts the questions shadow traffic and A/B testing answer and explains why teams can use them in that order.

Key terms: shadow traffic, A/B testing, technical readiness, performance requirements, business impact

4.3.3 Benefits and challenges of shadow traffic

This section explains what shadow traffic can reveal about a new AI system under real production traffic and the infrastructure, safety, and privacy challenges it creates.

Key terms: shadow traffic, production system, realistic requests, quality comparisons, production data patterns, bottlenecks, request processing pipeline, extended shadow testing, infrastructure costs, computational load

4.3.4 Mimicking shadow traffic

The section sketches a simplified shadow-traffic simulation for comparing production and shadow-system latency against performance thresholds.

Key terms: shadow traffic router, latency distributions, performance thresholds

4.4 Latency degradation experiments

This section introduces latency degradation experiments as a way to measure the impact of slower experiences while limiting harm to users.

Key terms: latency degradation experiment, controlled latency, A/B test, limited exposure, short duration, predefined stop conditions, guardrail metrics, stakeholder approval, latency threshold

4.4.1 4.4.1 Designing latency degradation tests

This section explains how to design latency degradation A/B tests that isolate the effects of slower experiences on product and user metrics.

Key terms: latency degradation, A/B test, test variant, configurable delay, device type, targeting criteria, test environment, production environment, controlled delays, system resources

4.4.2 Key online evaluation system performance metrics

This section explains which system performance metrics to track in a latency degradation experiment and how they reveal user-facing slowdowns.

Key terms: response time, end-to-end latency, baseline, bottleneck, model inference time, memoization, quantization, pruning, distillation, application load time

4.4.3 Movie recommendations example

The section uses a movie-recommendations A/B test to show how measured latency costs can inform model-launch decisions.

Key terms: latency degradation, A/B test, p99, control group, experiment cells, device type, page load time percentiles, CTR, bounce rate, abandonment rate

4.5 Engineering considerations

This section explains how cross-layer monitoring, meaningful alerts, and rollback triggers help evaluate and protect the production performance of AI features.

Key terms: unified dashboard, system metrics, performance regressions, alert thresholds, historical baselines, static, relative, automated rollback triggers, predefined threshold, feature flag

4.5.1 4.5.1 Make sure to capacity plan your infrastructure resources

This subsection explains how to plan production infrastructure capacity using load tests, cost and scaling tradeoffs, and continued bottleneck measurement even when autoscaling is available.

Key terms: CPU, memory, GPU, load testing, request volume, Cost-Performance Tradeoffs, vertical scaling, horizontal scaling, autoscaling, bottlenecks

4.5.2 4.5.2 You’ll never regret time spent testing under realistic product conditions

This section explains how to test AI system performance under realistic production conditions rather than evaluating model inference alone.

Key terms: production environments, spiky, unpredictable patterns, seasonal variations, event-driven traffic surges, edge cases, complete request flow, user interaction, response delivery, model inference

4.5.3 4.5.3 Keep in mind these performance optimization tactics

This subsection lists ways to improve system performance after identifying bottlenecks, including model, request-processing, hardware, and agent-level optimizations.

Key terms: quantization, pruning, knowledge distillation, ONNX, hardware acceleration, batch size, throughput, latency, feature preprocessing, prompt tokens

4.6 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 3: Using offline evaluations as diagnostics Chapter 5: Counterfactual evaluations →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.