Chapter 4: Engineering system performance evaluations
This introduction explains why evaluating an AI feature requires assessing the production system around the model, not just the model’s offline performance.
After this chapter you will be able to
- Production Performance Evaluation Purpose. Explain why strong offline model results do not establish that an AI feature will work well in production, and distinguish engineering system performance evaluations from diagnostic evaluations.
- Production performance beyond model quality. Explain why a high-performing model is not sufficient for a user-facing product: the surrounding system must serve predictions quickly and reliably under load.
- User impact of visible delays. Describe how slow loading or end-to-end latency can affect users, while noting that the size of the effect depends on the product and where the delay occurs.
- Inference time versus page load. Distinguish a change in backend model inference time from a change in total page load time, and explain why user visibility matters.
- Performance and business outcomes. Explain how slow pages or delayed AI features can reduce user engagement and affect business metrics, using the recommendation example.
- Visible and perceived latency. Distinguish real latency from perceived latency and explain why a user-visible delay may matter more than a hidden backend delay.
- Responsiveness in model evaluation. Justify including serving responsiveness alongside offline model quality when evaluating an AI-powered product.
- Production versus controlled performance. Explain why a model that performs well in development or offline evaluation may struggle in production, and why engineering performance should be measured through both offline and online evaluations.
- End-to-end request latency. Trace how work around a model can add latency across a production request, from obtaining inputs through preparing the output for users.
- Nonlinear capacity demands. Explain why serving substantially more AI requests may require more than a proportional increase in computing resources.
- Whole-system performance perspective. Explain why evaluating an AI system requires more than checking prediction quality, and describe what the main categories of engineering metrics reveal when considered together.
- User experience and system stack. Explain why load time matters to users and why a slow AI feature should not automatically be blamed on model inference.
- Stages of visible loading. Distinguish Time to First Byte, First Contentful Paint, and Fully Loaded Time by what each measures, and describe an improvement approach given for each.
- End-to-end response latency. Explain what end-to-end latency measures, which stages it includes, and why it should be monitored across the whole system.
- Model inference latency factors. Describe what model inference latency measures, what can make it vary, and the optimization options named in the section.
- Batch versus live inference. Distinguish batching from live inference and explain the responsiveness, throughput, and infrastructure trade-off used to choose between them.
- User-visible response timing. Explain how time to first token, tokens per second, and time to final token describe different parts of an LLM response, and why the first visible output can matter more to perceived speed than total response time.
- Agent task orchestration metrics. Explain why multi-step agents need task-level metrics, including model and tool calls, tool latency and reliability, step count, loops or retries, and external dependency failures. Describe how repeated steps make total task latency cumulative even when individual model calls are fast.
- Combined system performance metrics. Explain why a performance evaluation considers both initial load time and responsiveness during interaction, using metrics from the frontend, backend, AI components, and infrastructure to locate useful optimizations.
- Quality and performance tradeoffs. Explain why an improvement to speed or throughput can come at the expense of another outcome, and describe how model choice must balance product quality with system requirements.
- Shadow traffic setup. Explain that shadow traffic sends a copy of production traffic through a candidate system while users continue to receive responses from the current production system.
- User impact boundary. Distinguish the production-readiness question shadow traffic can answer from the user-behavior question that requires an online evaluation such as an A/B test.
- Shadow request routing. Trace how a user request reaches both the production system and a shadow system, and explain which response the user receives.
- Technical readiness before business impact. Distinguish the question shadow traffic answers from the question A/B testing answers, and explain why a team might run them sequentially for a new AI model.
- Realistic Low Risk Comparisons. Explain how shadow traffic supports performance and quality comparisons using real requests without showing experimental responses to users.
- End to End Production Confidence. Describe why observing the full request pipeline across changing production traffic can reveal problems and build confidence before a launch.
- Infrastructure Load And Cost. Explain the infrastructure and capacity costs of duplicating requests, and describe how goal-driven sampling or throttling can limit them.
- Shadow Side Effect Isolation. Describe why state-changing and externally acting shadow requests need isolation or interception, including for LLM agents.
- Shadow traffic performance simulation. Describe how the simplified simulation sends each request to both production and shadow systems, compares their latency distributions, and uses predefined thresholds to inform decisions.
- Controlled latency degradation. Explain how a latency degradation experiment tests whether slower performance affects users: an A/B test intentionally adds controlled latency and measures changes in user, product, and business metrics.
- User protection and decision purpose. Justify why latency degradation experiments need safeguards, and describe how their results can inform future model and system decisions.
- Controlled latency test variants. Describe how to add configurable delays in an A/B test and use variants with different delay amounts to find where product and user metrics begin to worsen.
- Production-like test resources. Justify matching test infrastructure and system resources to production conditions when simulating latency.
- End-to-end response time. Explain what end-to-end response time measures and why a team should establish a baseline and break it down by stage before optimizing latency.
- Tail latency percentiles. Explain what p90 and p99 response-time percentiles show and why slow-tail requests matter even when most requests perform well.
- Latency degradation test design. Describe how the example adds backend delay to the recommendations page, compares a control with five delayed variants, and repeats the test by device type. Explain why it uses 100 ms rather than 10 ms increments.
- System and user metrics. Explain why the test tracks page-load percentiles alongside user behaviors such as clicks, departures, and downstream movie consumption. Distinguish bouncing after load from abandoning before load completes.
- Mobile latency launch guardrail. Interpret the reported mobile performance cliff between 200 ms and 300 ms of added latency and explain how it informs decisions about slower model iterations.
- Cross-layer system monitoring. Explain why evaluating an AI model iteration requires monitoring the system that delivers it, not just the model, and how a unified dashboard helps locate performance bottlenecks.
- Baseline-based alert thresholds. Describe how teams can set monitoring alerts using historical baselines and product expectations, including the difference between static and relative thresholds.
- Resource needs under load. Explain how to estimate a model’s CPU, memory, and GPU requirements before deployment by testing at different traffic levels and observing how needs change as request volume grows.
- Autoscaling and bottleneck limits. Explain why autoscaling does not replace performance evaluation and where bottlenecks must continue to be measured.
- Production-like test conditions. Describe how performance tests should reflect actual usage, including unpredictable traffic and the range of inputs the model will encounter.
- End-to-end request testing. Explain why performance tests should cover the complete request flow rather than model inference in isolation.
- Model and hardware optimization. Describe how model optimization techniques and specialized hardware can improve inference performance, while noting the aim of avoiding substantial accuracy loss.
- Batching and preprocessing trade-offs. Explain why batch size should be tested and how inefficient feature preprocessing can be improved.
- Full-path agent optimization. Describe the suggested LLM and agent performance tactics and explain why optimization should cover the full task path.
What it builds on
- Diagnosing model behavior (chapter 3) · The distinct purpose of diagnosing model behavior.
Section by section
Introduction
This introduction explains why evaluating an AI feature requires assessing the production system around the model, not just the model’s offline performance.
- System Behind the Model: Describe how the engineering system turns a model prediction into a product experience by fetching data, invoking the model, serving its output, and handling failures.
Why it matters: The user experiences the whole serving system, not a prediction in isolation. - Production Performance Evaluation Purpose: Explain why strong offline model results do not establish that an AI feature will work well in production, and distinguish engineering system performance evaluations from diagnostic evaluations.
Why it matters: A promising model is useful only if the feature can deliver an acceptable experience under production conditions.
Key terms: engineering system, product experience, engineering system performance evaluations, offline diagnostic evaluations, reliably, quickly, cost-effectively
4.1 Why latency and load time really matter
This section explains why a production AI feature must be served quickly and reliably, and why the user-visible effect of latency matters more than model speed alone.
- Production performance beyond model quality: Explain why a high-performing model is not sufficient for a user-facing product: the surrounding system must serve predictions quickly and reliably under load.
Why it matters: Users experience the whole product, not its offline model score. - User impact of visible delays: Describe how slow loading or end-to-end latency can affect users, while noting that the size of the effect depends on the product and where the delay occurs.
Why it matters: Performance evaluations need to account for delays users may actually feel. - Inference time versus page load: Distinguish a change in backend model inference time from a change in total page load time, and explain why user visibility matters.
Why it matters: A backend timing change needs to be interpreted in terms of what users actually experience. - Fallbacks as safety mechanisms: Explain the intended role of fallbacks when a model or service is unavailable, rather than treating them as the normal AI-powered experience.
Why it matters: A safe backup does not replace making the intended feature dependable.
Key terms: offline quality, under load, scalable engineering architecture, end-to-end latency, user engagement, conversion, backend model inference time, total page load time, model-serving latency, progressive rendering
4.1.1 Illustrating the impact on user experience and business metrics
This section explains how user-visible system delays affect product and business outcomes and why model evaluations must account for responsiveness as well as offline quality.
- Performance and business outcomes: Explain how slow pages or delayed AI features can reduce user engagement and affect business metrics, using the recommendation example.
Why it matters: A performance delay can change the product outcomes an evaluation is meant to protect. - Visible and perceived latency: Distinguish real latency from perceived latency and explain why a user-visible delay may matter more than a hidden backend delay.
Why it matters: Measuring only total response time can miss how fast or slow the product feels to users. - Responsiveness in model evaluation: Justify including serving responsiveness alongside offline model quality when evaluating an AI-powered product.
Why it matters: A model choice that looks best offline may fail when users have to wait for its predictions.
Key terms: system performance, user engagement, page views, impressions, Ad revenue, real latency, perceived latency, streaming, time to the first token, accuracy
4.1.2 Technical constraints and scaling challenges
Production AI performance can differ sharply from offline results because real traffic, end-to-end request dependencies, and scaling trade-offs create latency that isolated model tests may miss.
- Production versus controlled performance: Explain why a model that performs well in development or offline evaluation may struggle in production, and why engineering performance should be measured through both offline and online evaluations.
Why it matters: Good offline performance alone does not establish that users will experience acceptable performance. - End-to-end request latency: Trace how work around a model can add latency across a production request, from obtaining inputs through preparing the output for users.
Why it matters: Optimizing model inference alone may leave the delays users actually experience untouched. - Nonlinear capacity demands: Explain why serving substantially more AI requests may require more than a proportional increase in computing resources.
Why it matters: Traffic growth can expose capacity problems that a simple resource estimate would miss. - Batching throughput trade-off: Describe the trade-off introduced by batching requests and why batch size needs tuning.
Why it matters: A system can handle more total work yet deliver slower responses to individual users.
Key terms: controlled environment, production traffic, steady-state testing, data distribution shifts, offline and online evaluations, data normalization, feature engineering, text tokenization, data dependencies, middleware overhead
4.2 Engineering system performance metrics
This section introduces the categories of engineering performance metrics and explains why they should be considered together alongside prediction quality.
- Whole-system performance perspective: Explain why evaluating an AI system requires more than checking prediction quality, and describe what the main categories of engineering metrics reveal when considered together.
Why it matters: A model’s good predictions do not by themselves ensure that the product delivers a good experience.
Key terms: frontend load time metrics, backend latency metrics, model-serving metrics, infrastructure metrics, orchestration metrics
4.2.1 Key load time metrics
This section explains how user-visible load time and model-serving readiness affect AI product performance, and introduces metrics for measuring different parts of that experience.
- User experience and system stack: Explain why load time matters to users and why a slow AI feature should not automatically be blamed on model inference.
Why it matters: Investigating the whole stack helps you improve what users actually experience, rather than optimizing the wrong component. - Stages of visible loading: Distinguish Time to First Byte, First Contentful Paint, and Fully Loaded Time by what each measures, and describe an improvement approach given for each.
Why it matters: These metrics locate different delays between a request and a complete, visible AI-powered page. - User-centered web performance metrics: Describe what LCP, INP, and CLS capture and why they can matter for AI-powered web products.
Why it matters: AI-generated page elements can change both what users see and how stable and responsive the page feels. - Model readiness versus page rendering: Distinguish model-serving lifecycle metrics from page-rendering metrics and describe when serving readiness can affect users.
Why it matters: A page can be delayed by serving readiness even though readiness and on-screen rendering are different things to measure.
Key terms: load time metrics, inference latency, CDNs, stack, Time to First Byte, TTFB, First Contentful Paint, FCP, Fully Loaded Time, progressive rendering
4.2.2 Key latency metrics
This section defines two latency metrics for interactive AI systems and explains the factors and serving choices that affect responsiveness.
- End-to-end response latency: Explain what end-to-end latency measures, which stages it includes, and why it should be monitored across the whole system.
Why it matters: Users experience the wait for the complete response, not just the model’s computation time. - Model inference latency factors: Describe what model inference latency measures, what can make it vary, and the optimization options named in the section.
Why it matters: Isolating model time helps teams identify which delays they can address through model or infrastructure choices. - Batch versus live inference: Distinguish batching from live inference and explain the responsiveness, throughput, and infrastructure trade-off used to choose between them.
Why it matters: A serving approach that improves throughput may still be a poor fit if users need an immediate response.
Key terms: end-to-end latency, Application Performance Monitoring (APM), Model Inference Latency, quantization, pruning, distillation, hardware accelerators, batching, live inference, throughput
4.2.3 LLM and agent-specific performance metrics
This section identifies LLM and agent-specific performance metrics that reveal how quickly and reliably a user receives a response or completes a task.
- User-visible response timing: Explain how time to first token, tokens per second, and time to final token describe different parts of an LLM response, and why the first visible output can matter more to perceived speed than total response time.
Why it matters: A response can feel responsive or sluggish before it is complete. - LLM pipeline cost and reliability: Describe the LLM performance measures beyond visible generation: prompt processing time, context size, retrieval latency, cost per request, and timeout, retry, and fallback rates.
Why it matters: Tracking work around the model call helps reveal costs, delays, and requests that do not complete cleanly. - Agent task orchestration metrics: Explain why multi-step agents need task-level metrics, including model and tool calls, tool latency and reliability, step count, loops or retries, and external dependency failures. Describe how repeated steps make total task latency cumulative even when individual model calls are fast.
Why it matters: A fast model call does not guarantee a fast or reliable completed agent task.
Key terms: time to first token (TTFT), tokens per second, time to final token, prompt processing time, context size, retrieval latency, cost per request, timeout, retry, and fallback rates, one-shot agent, multi-step agent
4.2.4 Combining load time and latency metrics for performance evaluations
This section shows how to evaluate AI system performance by combining load-time and latency metrics with model quality and product requirements.
- Combined system performance metrics: Explain why a performance evaluation considers both initial load time and responsiveness during interaction, using metrics from the frontend, backend, AI components, and infrastructure to locate useful optimizations.
Why it matters: A slow experience may have causes in different parts of the system, so one metric alone may not point to the best fix. - Quality and performance tradeoffs: Explain why an improvement to speed or throughput can come at the expense of another outcome, and describe how model choice must balance product quality with system requirements.
Why it matters: Optimizing a single performance number can make the overall product worse.
Key terms: load time metrics, latency metrics, frontend performance metrics, backend service metrics, AI-specific metrics, infrastructure metrics, model quality, product impact, inference latency, throughput
4.3 Shadow traffic
Introduces shadow traffic as a way to test a candidate system under real production traffic without showing its output to users, and explains what this can and cannot evaluate.
- Shadow traffic setup: Explain that shadow traffic sends a copy of production traffic through a candidate system while users continue to receive responses from the current production system.
Why it matters: This setup makes it possible to test a candidate under real traffic without changing what users see. - Production performance evidence: Describe what shadow traffic can measure and why real traffic patterns may expose bottlenecks that synthetic tests miss.
Why it matters: Real traffic can uncover production bottlenecks that controlled test traffic overlooks. - User impact boundary: Distinguish the production-readiness question shadow traffic can answer from the user-behavior question that requires an online evaluation such as an A/B test.
Why it matters: A technically ready system is not necessarily an experience users will prefer.
Key terms: shadow traffic, dark loading, production traffic, candidate system, latency, throughput, error rates, resource utilization, fallback behavior, output differences
4.3.1 How shadow traffic works
This section explains how shadow traffic sends copies of real user requests to a new system for evaluation while users continue to receive responses from the production system.
- Shadow request routing: Trace how a user request reaches both the production system and a shadow system, and explain which response the user receives.
Why it matters: This routing lets a team try a new component on real requests without changing what users receive. - Real traffic comparison logging: Describe what teams capture from production and shadow responses and why real traffic is useful for evaluating a new model or component.
Why it matters: The comparison provides evidence about the new system under realistic traffic while preserving the existing user experience.
Key terms: shadow traffic, request duplication, traffic routing, production system, shadow system, detailed logging, real production traffic, latency, throughput, resource utilization
4.3.2 Shadow traffic versus A/B testing
This section contrasts the questions shadow traffic and A/B testing answer and explains why teams can use them in that order.
- Technical readiness before business impact: Distinguish the question shadow traffic answers from the question A/B testing answers, and explain why a team might run them sequentially for a new AI model.
Why it matters: Checking performance first makes the subsequent business-impact test easier to interpret and less risky for users.
Key terms: shadow traffic, A/B testing, technical readiness, performance requirements, business impact
4.3.3 Benefits and challenges of shadow traffic
This section explains what shadow traffic can reveal about a new AI system under real production traffic and the infrastructure, safety, and privacy challenges it creates.
- Realistic Low Risk Comparisons: Explain how shadow traffic supports performance and quality comparisons using real requests without showing experimental responses to users.
Why it matters: It lets teams examine a new system against real requests before its responses reach users. - End to End Production Confidence: Describe why observing the full request pipeline across changing production traffic can reveal problems and build confidence before a launch.
Why it matters: A model can look sound in isolation yet encounter problems elsewhere in the pipeline or under changing demand. - Infrastructure Load And Cost: Explain the infrastructure and capacity costs of duplicating requests, and describe how goal-driven sampling or throttling can limit them.
Why it matters: A test intended to protect production can itself degrade production if its extra load is not planned for. - Shadow Side Effect Isolation: Describe why state-changing and externally acting shadow requests need isolation or interception, including for LLM agents.
Why it matters: A shadow request can still cause real-world harm if its actions are executed rather than contained. - Duplicated Data Security: Explain the security and retention concern created when shadow systems process copies of user requests.
Why it matters: Keeping experimental responses hidden does not remove the privacy obligations of processing copied requests.
Key terms: shadow traffic, production system, realistic requests, quality comparisons, production data patterns, bottlenecks, request processing pipeline, extended shadow testing, infrastructure costs, computational load
4.3.4 Mimicking shadow traffic
The section sketches a simplified shadow-traffic simulation for comparing production and shadow-system latency against performance thresholds.
- Shadow traffic performance simulation: Describe how the simplified simulation sends each request to both production and shadow systems, compares their latency distributions, and uses predefined thresholds to inform decisions.
Why it matters: This shows what the example is meant to compare without mistaking it for an evaluation of recommendation quality.
Key terms: shadow traffic router, latency distributions, performance thresholds
4.4 Latency degradation experiments
This section introduces latency degradation experiments as a way to measure the impact of slower experiences while limiting harm to users.
- Controlled latency degradation: Explain how a latency degradation experiment tests whether slower performance affects users: an A/B test intentionally adds controlled latency and measures changes in user, product, and business metrics.
Why it matters: The experiment gives teams evidence about how sensitive users are to slower performance. - User protection and decision purpose: Justify why latency degradation experiments need safeguards, and describe how their results can inform future model and system decisions.
Why it matters: Safeguards reduce the cost to users while the team learns which performance trade-offs matter.
Key terms: latency degradation experiment, controlled latency, A/B test, limited exposure, short duration, predefined stop conditions, guardrail metrics, stakeholder approval, latency threshold
4.4.1 4.4.1 Designing latency degradation tests
This section explains how to design latency degradation A/B tests that isolate the effects of slower experiences on product and user metrics.
- Controlled latency test variants: Describe how to add configurable delays in an A/B test and use variants with different delay amounts to find where product and user metrics begin to worsen.
Why it matters: The results can ground decisions about whether slower AI-powered features need optimization or are acceptable given business goals. - Device-specific experiment isolation: Explain why and how to run separate latency degradation tests for different device types when users can access the product on multiple devices.
Why it matters: Separating devices makes the results easier to interpret when their implementations differ. - Production-like test resources: Justify matching test infrastructure and system resources to production conditions when simulating latency.
Why it matters: Production-like resources help ensure the experiment measures the effect of the intended delay rather than an infrastructure bottleneck.
Key terms: latency degradation, A/B test, test variant, configurable delay, device type, targeting criteria, test environment, production environment, controlled delays, system resources
4.4.2 Key online evaluation system performance metrics
This section explains which system performance metrics to track in a latency degradation experiment and how they reveal user-facing slowdowns.
- End-to-end response time: Explain what end-to-end response time measures and why a team should establish a baseline and break it down by stage before optimizing latency.
Why it matters: A slow user experience may come from any part of the serving system, not just the model. - Model inference time: Distinguish model inference time from total response time and describe ways this section says inference can be sped up, including avoiding repeated computation.
Why it matters: Shortening inference helps overall latency when model computation is a substantial source of delay. - Application load time: Describe what application load time includes and how a lightweight initial experience or deferred content can make an AI-driven application usable sooner.
Why it matters: Users may abandon a page before interacting if loading and personalization keep it from becoming usable. - Resource utilization guardrails: Describe what CPU, memory, and network utilization reveal and how the section says to use these metrics in a latency degradation experiment.
Why it matters: Resource measurements help reveal how the system behaves under load alongside user-facing timing metrics. - Tail latency percentiles: Explain what p90 and p99 response-time percentiles show and why slow-tail requests matter even when most requests perform well.
Why it matters: An apparently fast system can still give its slowest requests a poor or degraded experience.
Key terms: response time, end-to-end latency, baseline, bottleneck, model inference time, memoization, quantization, pruning, distillation, application load time
4.4.3 Movie recommendations example
The section uses a movie-recommendations A/B test to show how measured latency costs can inform model-launch decisions.
- Latency degradation test design: Describe how the example adds backend delay to the recommendations page, compares a control with five delayed variants, and repeats the test by device type. Explain why it uses 100 ms rather than 10 ms increments.
Why it matters: This design makes the cost of a slower recommendations experience measurable rather than a matter of intuition. - System and user metrics: Explain why the test tracks page-load percentiles alongside user behaviors such as clicks, departures, and downstream movie consumption. Distinguish bouncing after load from abandoning before load completes.
Why it matters: A slower page is a product concern when its technical slowdown translates into changed user behavior. - Mobile latency launch guardrail: Interpret the reported mobile performance cliff between 200 ms and 300 ms of added latency and explain how it informs decisions about slower model iterations.
Why it matters: The measured cliff gives teams a concrete way to weigh a model-quality gain against its latency cost.
Key terms: latency degradation, A/B test, p99, control group, experiment cells, device type, page load time percentiles, CTR, bounce rate, abandonment rate
4.5 Engineering considerations
This section explains how cross-layer monitoring, meaningful alerts, and rollback triggers help evaluate and protect the production performance of AI features.
- Cross-layer system monitoring: Explain why evaluating an AI model iteration requires monitoring the system that delivers it, not just the model, and how a unified dashboard helps locate performance bottlenecks.
Why it matters: A model can perform well in isolation while the feature that serves it remains slow or unreliable. - Baseline-based alert thresholds: Describe how teams can set monitoring alerts using historical baselines and product expectations, including the difference between static and relative thresholds.
Why it matters: Meaningful thresholds help teams notice performance changes that warrant investigation. - High-risk release rollback triggers: Explain how monitoring can protect a high-risk release by triggering automated traffic reduction, feature disabling, or rollback when predefined metrics cross thresholds.
Why it matters: Automated responses can limit user exposure when a new AI feature causes an operational regression.
Key terms: unified dashboard, system metrics, performance regressions, alert thresholds, historical baselines, static, relative, automated rollback triggers, predefined threshold, feature flag
4.5.1 4.5.1 Make sure to capacity plan your infrastructure resources
This subsection explains how to plan production infrastructure capacity using load tests, cost and scaling tradeoffs, and continued bottleneck measurement even when autoscaling is available.
- Resource needs under load: Explain how to estimate a model’s CPU, memory, and GPU requirements before deployment by testing at different traffic levels and observing how needs change as request volume grows.
Why it matters: Load testing helps you plan enough infrastructure for the traffic the model will face. - Infrastructure scaling tradeoffs: Describe how to weigh latency gains against infrastructure cost and choose between more powerful machines and more machines according to the model’s characteristics.
Why it matters: The fastest setup is not necessarily the best business or technical choice. - Autoscaling and bottleneck limits: Explain why autoscaling does not replace performance evaluation and where bottlenecks must continue to be measured.
Why it matters: Adding capacity may leave the real cause of a slow product experience untouched.
Key terms: CPU, memory, GPU, load testing, request volume, Cost-Performance Tradeoffs, vertical scaling, horizontal scaling, autoscaling, bottlenecks
4.5.2 4.5.2 You’ll never regret time spent testing under realistic product conditions
This section explains how to test AI system performance under realistic production conditions rather than evaluating model inference alone.
- Production-like test conditions: Describe how performance tests should reflect actual usage, including unpredictable traffic and the range of inputs the model will encounter.
Why it matters: A test that omits traffic surges or difficult inputs may miss performance problems users will experience. - End-to-end request testing: Explain why performance tests should cover the complete request flow rather than model inference in isolation.
Why it matters: Users experience the response delivered by the whole system, not the model’s inference time alone.
Key terms: production environments, spiky, unpredictable patterns, seasonal variations, event-driven traffic surges, edge cases, complete request flow, user interaction, response delivery, model inference
4.5.3 4.5.3 Keep in mind these performance optimization tactics
This subsection lists ways to improve system performance after identifying bottlenecks, including model, request-processing, hardware, and agent-level optimizations.
- Model and hardware optimization: Describe how model optimization techniques and specialized hardware can improve inference performance, while noting the aim of avoiding substantial accuracy loss.
Why it matters: These are options to consider when the bottleneck is in model inference. - Batching and preprocessing trade-offs: Explain why batch size should be tested and how inefficient feature preprocessing can be improved.
Why it matters: Improving throughput alone may not improve the wait experienced by an individual requester. - Full-path agent optimization: Describe the suggested LLM and agent performance tactics and explain why optimization should cover the full task path.
Why it matters: An agent can remain slow even if an individual model call is fast.
Key terms: quantization, pruning, knowledge distillation, ONNX, hardware acceleration, batch size, throughput, latency, feature preprocessing, prompt tokens
4.6 Summary
Where this is used later
- Balancing system performance and logging (chapter 5) · Why latency matters to the user experience
- Infrastructure-induced latency or errors (chapter 6) · Why latency and load time affect user experience and business metrics.
- Model selection playbook (chapter 7) · Informs model selection when quality must be weighed against product responsiveness.
- Robustness in edge cases (chapter 8) · The previously introduced request-mirroring approach to shadow traffic.
- Designing an A/B test (chapter 6) · A/B test design builds on the distinction between checking technical readiness and measuring the impact of a change.
- Latency monitoring (chapter 6) · Latency degradation tests can quantify how latency affects product and user metrics.
- Translating offline evaluations into online watch points (chapter 7) · Supports translating performance findings into online watch points.
- Realtime monitoring tied to product and business metrics (chapter 7) · Supports monitoring tied to product expectations and operational signals.
Concepts explained in this chapter
- frontend load time metricThe elapsed time required for a web page to load, often summarized across request-time percentiles.
- latencyThe total elapsed time from receiving a user request to displaying the complete response, including all intermediate processing steps.
- latency degradation experimentA controlled evaluation that deliberately increases or varies response latency to measure effects on system, user, product, or business outcomes.
- shadow testingA technique that sends copies of live production requests to a non-serving candidate system and records its outputs without affecting user-visible behavior.
- fallbackAn alternative system behavior or response used when a preferred processing path or primary component is unavailable or fails.
- batchingThe grouping of multiple requests or data items into batches for joint processing.
- distillationA model compression technique in which a smaller student model learns to reproduce the behavior of a larger teacher model.
- pruningA model optimization technique that removes parameters, connections, or structures judged unnecessary for the target task or inference.
- quantizationA model optimization technique that represents model values with lower-precision numerical formats to reduce computation, memory use, or latency.
- resource utilizationHardware utilization is the proportion of available processor capacity consumed by a running system.
- capacity planningCapacity planning is the process of determining the computational resources required to operate a system under expected and peak workloads.
- time to first tokenTime to first token is the elapsed time from a request until the first generated token becomes available to the user.
- percentileA latency percentile is a value below which a specified proportion of observed system response times falls.
- performanceA performance metric is a quantitative measure of how well a model or system achieves its intended task or objective.
- throughputTokens per second is the rate at which a generative model produces output tokens after generation begins.
- A/B testA controlled online experiment that compares outcomes between two variants, typically a control and a treatment, using assigned users or traffic.
- agent step countThe number of execution steps performed while an agent system completes a task.
- context windowA context window is the bounded amount of input and generated text that a language model can process in one invocation.
- costCost measures the computational, financial, or resource expenditure required to operate a model or system.
- loop/retry rateThe frequency with which an agent system repeats an execution loop during task completion.
- time to final tokenTime to final token is the elapsed time from a request until the complete generated response is available.
- dry-run tool adapterA dry-run tool adapter is an integration layer that simulates a tool invocation without performing its externally visible side effects.
- network throttlingThrottling is the deliberate limitation of request or traffic processing rates to control resource use.
- samplingThe process of selecting a subset of observations or traffic from a larger population for analysis or processing.
- sink componentA sink component is a system component that captures outbound actions without delivering them to their external destination.
- statefulnessA stateful operation is an operation whose effects depend on or modify persistent application or external system state.
- reliabilityThe ability of a system to provide correct and available service consistently under expected operating conditions and failures.
- first contentful paintFirst Contentful Paint measures when the first content is rendered on a screen.
- fully loaded timeFully loaded time measures the elapsed time until a page and its content have completed loading.
- time to first byteTime to first byte measures the elapsed time between a client's request and receipt of the first byte of response data.
- metricA quantitative measure of model performance derived from behavior or outcomes observed during live operation.
- click-through rateClick-through rate is the proportion of users or impressions that result in a click on a presented item or recommendation.
- guardrail metricA guardrail metric is a metric monitored to ensure that an experiment does not cause unacceptable harm while optimizing its primary objective.
- monitoringThe continuous observation of model behavior, system signals, and user outcomes during live operation.
- user experienceUser experience is the overall experience a user has when interacting with a product, including its usefulness, quality, and responsiveness.
- evaluationA performance evaluation measures how well a model or AI system achieves its intended objective against predefined metrics.
- cachingThe storage of previously computed or retrieved data so subsequent requests can be served more quickly.
- progressive renderingProgressive rendering is the incremental display of interface content as portions become available rather than waiting for the entire result.
- memory usageMemory footprint is the amount of memory occupied by a model or software process during operation.
- control groupA control variant is the reference condition in an experiment against which treatment variants are compared.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
