What will I learn in chapter 9 of Building Reliable AI Systems?

The chapter introduction explains why increasingly capable LLM and agent systems need attention to both silent hallucinations and performance under production load. After reading it you should be able to: Explain how added capabilities introduce new failure points and why hallucinations are especially difficult to catch. Explain why a multi-agent system that works in testing can become slow, costly, and inaccurate when traffic rises, using the e-commerce example. Explain why hallucinations are difficult to detect: an LLM can present false information in the same confident tone as accurate information, such as invented product specifications or nonexistent legal cases. Explain how to establish grounding data as the benchmark for checking a chatbot’s factual claims, with coverage broad enough for its topics and facts specific enough to verify. Distinguish the generic test set of common inquiries from the adversarial test set of challenging or ambiguous inquiries, and describe how each is assembled. Describe how to extract verifiable factual claims from test responses and check them against grounding data, using automated comparisons and human review where appropriate. Explain what grounding defect rate, hallucination severity score, and topicwise analysis each report and how their results guide monitoring or improvement. Explain how FActScore breaks a response into atomic facts, checks each against grounding data, and reports the share that are correct. Contrast this with treating the whole response as grounded or hallucinated.

What does section 9.1, Identifying and measuring hallucinations, cover?

This section explains why confident but incorrect LLM outputs are hard to spot and why measuring hallucinations is necessary before addressing them. Explain why hallucinations are difficult to detect: an LLM can present false information in the same confident tone as accurate information, such as invented product specifications or nonexistent legal cases. Explain why systematically addressing hallucinations requires first finding them, which in turn requires a framework for measurement.

What does section 9.1.1, Four steps to identify and measure hallucinations, cover?

This section presents a four-step process for measuring chatbot hallucinations: establish authoritative grounding data, build representative and adversarial tests, validate factual claims, and report actionable metrics. Explain how to establish grounding data as the benchmark for checking a chatbot’s factual claims, with coverage broad enough for its topics and facts specific enough to verify. Distinguish the generic test set of common inquiries from the adversarial test set of challenging or ambiguous inquiries, and describe how each is assembled. Describe how to extract verifiable factual claims from test responses and check them against grounding data, using automated comparisons and human review where appropriate.

What does section 9.1.2, FActScore: fine-grained factual evaluation, cover?

This section explains how FActScore evaluates individual factual claims in a response and notes the cost of that finer-grained assessment. Explain how FActScore breaks a response into atomic facts, checks each against grounding data, and reports the share that are correct. Contrast this with treating the whole response as grounded or hallucinated. Describe the tradeoff between FActScore’s granular accuracy measurement and its need for LLM calls to extract and verify facts; state when simpler text-overlap metrics can provide a faster signal.

What does section 9.1.3, ROUGE for summarization evaluation, cover?

This section explains how ROUGE compares generated text with reference text for summarization evaluation and why word overlap alone cannot establish factual accuracy. Explain that ROUGE measures overlap between generated text and a reference output, making it useful when a test set contains expected responses. Distinguish ROUGE-1, ROUGE-2, and ROUGE-L by the kind of overlap each measures. Explain why neither a high nor a low ROUGE score conclusively establishes whether generated text is accurate or hallucinated.

What does section 9.1.4, LLM-as-a-judge: A holistic approach, cover?

This section explains how an LLM judge evaluates hallucinations, why its judgments can be unreliable, and how production systems check them. Explain how an LLM judge evaluates a generated response against a reference and why this can reveal inaccuracies that token-overlap metrics miss. Describe how a judge model's own errors and biases can make hallucination evaluations unreliable. Describe the complementary checks production systems use to reduce reliance on any one judge model.

What does section 9.1.5, Red teaming and stress testing, cover?

This section explains how red-teaming exposes LLM failures that standard evaluations may miss and uses repeated testing and refinement to improve reliability. Explain how a dedicated team tries to make an AI system fail to find unexpected vulnerabilities before users encounter them, and why this complements rather than replaces metrics and benchmarks. Trace the cycle of challenging an LLM with adversarial tests, extreme scenarios, and edge cases; analyzing the weaknesses found; improving the system; and testing it again. Explain the role and limit of using LLMs to generate adversarial examples and stress tests.

What does section 9.2, Essential architectural patterns for performance, cover?

This introduction explains why an LLM application's architecture matters for responsiveness, scalability, and cost before previewing performance patterns. Explain why connecting a prompt to an API is not enough for a production LLM application: architecture affects response speed, scaling under traffic, reuse of answers, and reliance on expensive models.

What does section 9.2.3, Caching for efficiency, cover?

This section explains how exact-match and semantic caching reduce repeated LLM calls while using similarity checks and cache invalidation to protect answer quality. Distinguish exact string matching from semantic caching, and explain why reusing responses to repeated or rephrased queries reduces latency and cost. Trace how a semantic cache stores a query and response, looks up a new query, and decides whether to return a stored answer or call the LLM. Explain why the closest cached query may still be a poor match and how a similarity threshold and LLM fallback prevent an irrelevant cached answer.

What should I know before reading chapter 9?

Multi-agent systems (chapter 8): Introduces multi-agent systems as an additional layer of capability.; Building intelligent agent coordination (chapter 8): Introduces coordination among agents in a multi-agent workflow.; Real-world example: ShopBot multi-agent system (chapter 8): The ShopBot example provides the e-commerce setting used for the grounding-data examples.; What are embeddings? (chapter 4): Introduces embeddings as representations that can be compared for meaning.; FAISS: Industry-standard vector indexing (chapter 4): Introduces FAISS as a vector indexing tool.; The cost-capability spectrum (chapter 2): Models differ in cost and capability.; Model selection as a reliability decision (chapter 2): Model selection affects application reliability.; Foundations of prompt engineering for reliable LLMs (chapter 2): Earlier prompt engineering provides the context for revising prompts while retaining their purpose.

Chapter 9: Evaluation and performance for LLMs and agents

What you will learn in chapter 9 of Building Reliable AI Systems

The chapter introduction explains why increasingly capable LLM and agent systems need attention to both silent hallucinations and performance under production load.

After this chapter you will be able to

Silent failures across system layers. Explain how added capabilities introduce new failure points and why hallucinations are especially difficult to catch.
Production load and agent overhead. Explain why a multi-agent system that works in testing can become slow, costly, and inaccurate when traffic rises, using the e-commerce example.
Confidently Incorrect LLM Outputs. Explain why hallucinations are difficult to detect: an LLM can present false information in the same confident tone as accurate information, such as invented product specifications or nonexistent legal cases.
Authoritative grounding dataset. Explain how to establish grounding data as the benchmark for checking a chatbot’s factual claims, with coverage broad enough for its topics and facts specific enough to verify.
Complementary measurement test sets. Distinguish the generic test set of common inquiries from the adversarial test set of challenging or ambiguous inquiries, and describe how each is assembled.
Claim extraction and validation. Describe how to extract verifiable factual claims from test responses and check them against grounding data, using automated comparisons and human review where appropriate.
Hallucination reporting metrics. Explain what grounding defect rate, hallucination severity score, and topicwise analysis each report and how their results guide monitoring or improvement.
Atomic fact accuracy scoring. Explain how FActScore breaks a response into atomic facts, checks each against grounding data, and reports the share that are correct. Contrast this with treating the whole response as grounded or hallucinated.
ROUGE Reference Text Comparison. Explain that ROUGE measures overlap between generated text and a reference output, making it useful when a test set contains expected responses.
Limits of Word Overlap. Explain why neither a high nor a low ROUGE score conclusively establishes whether generated text is accurate or hallucinated.
Reference-based LLM judging. Explain how an LLM judge evaluates a generated response against a reference and why this can reveal inaccuracies that token-overlap metrics miss.
Fallibility of judge models. Describe how a judge model's own errors and biases can make hallucination evaluations unreliable.
Checks on automated judgments. Describe the complementary checks production systems use to reduce reliance on any one judge model.
Purpose of red-teaming. Explain how a dedicated team tries to make an AI system fail to find unexpected vulnerabilities before users encounter them, and why this complements rather than replaces metrics and benchmarks.
Iterative red-teaming cycle. Trace the cycle of challenging an LLM with adversarial tests, extreme scenarios, and edge cases; analyzing the weaknesses found; improving the system; and testing it again.
Phoenix hallucination classification. Describe how Phoenix uses a judge LLM to compare a generated response with reference text, constrain the verdict to labels, and return a label and explanation for each data row.
Recurring hallucination rate alerts. Describe how the section proposes using classification results in production: evaluate a recurring sample of real outputs, track the hallucination rate over time, and alert when it exceeds a threshold.
Architecture and deployment performance. Explain why connecting a prompt to an API is not enough for a production LLM application: architecture affects response speed, scaling under traffic, reuse of answers, and reliance on expensive models.
Streaming versus complete replies. Distinguish revealing tokens as they are generated from waiting to show the entire reply, and explain why a live support chatbot might favor streaming while a formal document might favor a completed response.
System-Level Batching for Surges. Explain how an application can briefly queue incoming queries, dispatch them in groups, and smooth traffic without combining prompts into one GPT request. Describe the possible waiting-time tradeoff.
Offline Batch API Workflow. Trace how a large offline job is submitted through the OpenAI Batch API, from a request file to retrieved results. Explain its cost and turnaround tradeoff.
Choosing Complementary Batching Patterns. Distinguish when to use system-level batching versus the Batch API, including how both can serve different workloads in one system.
Exact and semantic cache matching. Distinguish exact string matching from semantic caching, and explain why reusing responses to repeated or rephrased queries reduces latency and cost.
Semantic cache request flow. Trace how a semantic cache stores a query and response, looks up a new query, and decides whether to return a stored answer or call the LLM.
Similarity thresholds and fallback. Explain why the closest cached query may still be a poor match and how a similarity threshold and LLM fallback prevent an irrelevant cached answer.
Stale answer prevention strategies. Explain why cached answers can become stale and describe ways to keep responses current.
Tiered model routing. Explain why a router sends routine queries to a cheaper, faster model and escalates complex or specialized queries to a more capable one.
Query complexity routing methods. Describe how heuristics, a trained classifier, or a smaller model using function calling can decide where to route a query, including the advantage of the smaller-model approach over keyword matching.
Two-pass quality escalation. Trace a two-pass fallback from a cheaper model’s first answer through a quality check to escalation, and explain its cost-versus-latency tradeoff.
Queued system-level batching. Trace how requests enter a shared queue, are collected into batches, and are dispatched concurrently to handle traffic surges.
Near-duplicate answer reuse. Explain how the service uses GPTCache to answer repeated or sufficiently similar questions without another real model call.
Complexity-based model routing. Describe how the service chooses between its everyday and advanced models for each request, and why.
More selective model fallback. Describe two ways to improve which model handles a query: a two-pass confidence check and a better complexity classifier.
Freshness of cached answers. Explain how versioning or timestamps can prevent cached answers from becoming misleading when store information changes.
Agent evaluation criteria. Distinguish typical standalone LLM evaluation from agent evaluation. Explain that an agent must be assessed not just for output correctness but also for its actions, tool interactions, decision-making efficiency, and execution path.
Pipeline and stepwise assessment. Explain why agent evaluation needs both an end-to-end assessment of the reasoning and execution pipeline and step-by-step analysis.
Limits of output-only evaluation. Explain why output quality metrics also apply to agents but cannot, on their own, reveal failures in an agent’s multistep work.
End-to-end agent evaluation. Explain why evaluating an agent requires more than judging its final text, and describe what a start-to-finish task test checks.
Hidden workflow errors. Explain why an apparently correct final output may still require stage-by-stage inspection, and name the stages this can help diagnose.
Execution trajectory comparison. Describe how comparing an execution trace with an expected trajectory helps evaluate and debug an agent’s workflow.

liveBook manning.com

Read chapter 9 on liveBook Get the book at manning.com

What it builds on

Multi-agent systems (chapter 8) · Introduces multi-agent systems as an additional layer of capability.
Building intelligent agent coordination (chapter 8) · Introduces coordination among agents in a multi-agent workflow.
Real-world example: ShopBot multi-agent system (chapter 8) · The ShopBot example provides the e-commerce setting used for the grounding-data examples.
What are embeddings? (chapter 4) · Introduces embeddings as representations that can be compared for meaning.
FAISS: Industry-standard vector indexing (chapter 4) · Introduces FAISS as a vector indexing tool.
The cost-capability spectrum (chapter 2) · Models differ in cost and capability.
Model selection as a reliability decision (chapter 2) · Model selection affects application reliability.
Foundations of prompt engineering for reliable LLMs (chapter 2) · Earlier prompt engineering provides the context for revising prompts while retaining their purpose.

Section by section

Introduction

The chapter introduction explains why increasingly capable LLM and agent systems need attention to both silent hallucinations and performance under production load.

Silent failures across system layers: Explain how added capabilities introduce new failure points and why hallucinations are especially difficult to catch.
Why it matters: An answer can look successful to the system while still giving a user false information.
Production load and agent overhead: Explain why a multi-agent system that works in testing can become slow, costly, and inaccurate when traffic rises, using the e-commerce example.
Why it matters: A successful demo does not show whether the system will remain responsive and accurate during a traffic surge.

Key terms: hallucination, retrieval system, agent, multi-agent architecture, orchestrator, cache, tool invocations

9.1 Identifying and measuring hallucinations

This section explains why confident but incorrect LLM outputs are hard to spot and why measuring hallucinations is necessary before addressing them.

Confidently Incorrect LLM Outputs: Explain why hallucinations are difficult to detect: an LLM can present false information in the same confident tone as accurate information, such as invented product specifications or nonexistent legal cases.
Why it matters: A confident-sounding answer is not, by itself, evidence that its claims are true.
Measurement Before Hallucination Fixes: Explain why systematically addressing hallucinations requires first finding them, which in turn requires a framework for measurement.
Why it matters: You need a way to identify the problem before you can tell what needs fixing.

Key terms: hallucinations, framework for measurement

9.1.1 Four steps to identify and measure hallucinations

This section presents a four-step process for measuring chatbot hallucinations: establish authoritative grounding data, build representative and adversarial tests, validate factual claims, and report actionable metrics.

Authoritative grounding dataset: Explain how to establish grounding data as the benchmark for checking a chatbot’s factual claims, with coverage broad enough for its topics and facts specific enough to verify.
Why it matters: Without a dependable source of truth, you cannot tell whether a response is grounded.
Complementary measurement test sets: Distinguish the generic test set of common inquiries from the adversarial test set of challenging or ambiguous inquiries, and describe how each is assembled.
Why it matters: Testing only routine questions can hide failures on ambiguous requests and edge cases.
Claim extraction and validation: Describe how to extract verifiable factual claims from test responses and check them against grounding data, using automated comparisons and human review where appropriate.
Why it matters: Checking specific assertions makes errors easier to locate than judging an entire answer by impression.
Hallucination reporting metrics: Explain what grounding defect rate, hallucination severity score, and topicwise analysis each report and how their results guide monitoring or improvement.
Why it matters: A single error count cannot show whether problems are severe, concentrated in one topic, or worsening after a change.

Key terms: grounding dataset, grounding data, authoritative information, generic test set, adversarial test set, measurement test sets, claims, potential hallucinations, human review, grounding defect rate (GDR)

9.1.2 FActScore: fine-grained factual evaluation

This section explains how FActScore evaluates individual factual claims in a response and notes the cost of that finer-grained assessment.

Atomic fact accuracy scoring: Explain how FActScore breaks a response into atomic facts, checks each against grounding data, and reports the share that are correct. Contrast this with treating the whole response as grounded or hallucinated.
Why it matters: Fact-level scoring shows how much of a partly inaccurate answer is still supported.
Granularity and evaluation cost: Describe the tradeoff between FActScore’s granular accuracy measurement and its need for LLM calls to extract and verify facts; state when simpler text-overlap metrics can provide a faster signal.
Why it matters: The extra detail of fact-level evaluation comes with a computational cost.

Key terms: FActScore, atomic facts, grounding data, GDR, LLM calls, reference responses, text-overlap metrics

9.1.3 ROUGE for summarization evaluation

This section explains how ROUGE compares generated text with reference text for summarization evaluation and why word overlap alone cannot establish factual accuracy.

ROUGE Reference Text Comparison: Explain that ROUGE measures overlap between generated text and a reference output, making it useful when a test set contains expected responses.
Why it matters: This clarifies when ROUGE can provide a useful evaluation signal.
Three ROUGE Similarity Variants: Distinguish ROUGE-1, ROUGE-2, and ROUGE-L by the kind of overlap each measures.
Why it matters: The variant tells you what kind of textual similarity a ROUGE score reflects.
Limits of Word Overlap: Explain why neither a high nor a low ROUGE score conclusively establishes whether generated text is accurate or hallucinated.
Why it matters: A word-overlap score should not be mistaken for a verdict on factual correctness.

Key terms: ROUGE, generated text, reference text, expected responses, ROUGE-1, unigram, ROUGE-2, bigram, ROUGE-L, longest common subsequence

9.1.4 LLM-as-a-judge: A holistic approach

This section explains how an LLM judge evaluates hallucinations, why its judgments can be unreliable, and how production systems check them.

Reference-based LLM judging: Explain how an LLM judge evaluates a generated response against a reference and why this can reveal inaccuracies that token-overlap metrics miss.
Why it matters: It offers a way to assess factual problems that text-overlap scores may overlook.
Fallibility of judge models: Describe how a judge model's own errors and biases can make hallucination evaluations unreliable.
Why it matters: An automated verdict is not trustworthy merely because another LLM produced it.
Checks on automated judgments: Describe the complementary checks production systems use to reduce reliance on any one judge model.
Why it matters: Independent checks help catch errors and biases in automated hallucination judgments.

Key terms: LLM-as-a-judge, reference, hallucinations, ROUGE, judge models, biases, subtle factual errors, multiple judge models, majority votes, human validation

9.1.5 Red teaming and stress testing

This section explains how red-teaming exposes LLM failures that standard evaluations may miss and uses repeated testing and refinement to improve reliability.

Purpose of red-teaming: Explain how a dedicated team tries to make an AI system fail to find unexpected vulnerabilities before users encounter them, and why this complements rather than replaces metrics and benchmarks.
Why it matters: A system can perform well on standard evaluations yet still fail in ways real users discover.
Iterative red-teaming cycle: Trace the cycle of challenging an LLM with adversarial tests, extreme scenarios, and edge cases; analyzing the weaknesses found; improving the system; and testing it again.
Why it matters: Retesting after improvements helps reveal whether the system has become more robust.
Oversight of generated stress tests: Explain the role and limit of using LLMs to generate adversarial examples and stress tests.
Why it matters: Generated test cases are useful only if they meaningfully challenge the system.

Key terms: red-teaming, adversarial testing, metrics and benchmarks, extreme scenarios, edge cases, continuous feedback loop, adversarial examples, stress tests, human oversight and validation

9.1.6 Using monitoring frameworks to detect hallucinations

The section explains why production hallucination detection needs monitoring and shows how Phoenix can classify outputs so teams can track and alert on hallucination rates.

Production monitoring pressures: Explain why development-time evaluation alone is insufficient for detecting hallucinations in production, citing scale, the need for quick detection, changing inputs, and integration with MLOps pipelines.
Why it matters: A production evaluation plan has to account for ongoing traffic and changing conditions, not just prelaunch tests.
Phoenix hallucination classification: Describe how Phoenix uses a judge LLM to compare a generated response with reference text, constrain the verdict to labels, and return a label and explanation for each data row.
Why it matters: Structured verdicts make the judge’s assessments usable for systematic monitoring rather than leaving them as freeform comments.
Recurring hallucination rate alerts: Describe how the section proposes using classification results in production: evaluate a recurring sample of real outputs, track the hallucination rate over time, and alert when it exceeds a threshold.
Why it matters: Repeated measurement turns individual hallucination judgments into a signal that can prompt action when output quality worsens.

Key terms: development-time evaluation, data drift, MLOps, Phoenix, HALLUCINATION_PROMPT_TEMPLATE, HALLUCINATION_PROMPT_RAILS_MAP, llm_ classify, judge LLM, reference text, results DataFrame

9.2 Essential architectural patterns for performance

This introduction explains why an LLM application's architecture matters for responsiveness, scalability, and cost before previewing performance patterns.

Architecture and deployment performance: Explain why connecting a prompt to an API is not enough for a production LLM application: architecture affects response speed, scaling under traffic, reuse of answers, and reliance on expensive models.
Why it matters: It explains why production performance depends on system design, not just the prompt or model API.

Key terms: LLM-powered application, scale, high-end models

9.2.1 Token streaming: Presenting answers incrementally or at the same time

This section compares streaming an answer as it is generated with showing a completed reply, explaining when each presentation choice helps and what it risks.

Streaming versus complete replies: Distinguish revealing tokens as they are generated from waiting to show the entire reply, and explain why a live support chatbot might favor streaming while a formal document might favor a completed response.
Why it matters: The right presentation choice depends on whether immediate feedback or a complete, polished answer matters more to the user.
Chunk-based streaming example: Describe how the section’s Python example produces a live-typing experience by consuming model output in chunks using an asynchronous function.
Why it matters: Consuming output in chunks connects the live-typing interface to how the example receives the model’s reply.

Key terms: token streaming, tokens, partial feedback, token-streaming, asynchronous function, chunks

9.2.2 Handling surges with batching: System-level vs. OpenAI’s Batch API

This section contrasts application-level batching for near-real-time traffic with OpenAI’s Batch API for large offline jobs and explains how to choose between them.

System-Level Batching for Surges: Explain how an application can briefly queue incoming queries, dispatch them in groups, and smooth traffic without combining prompts into one GPT request. Describe the possible waiting-time tradeoff.
Why it matters: This pattern can improve performance during bursts while keeping responses suitable for interactive users.
Offline Batch API Workflow: Trace how a large offline job is submitted through the OpenAI Batch API, from a request file to retrieved results. Explain its cost and turnaround tradeoff.
Why it matters: The workflow makes large jobs cheaper to run when immediate answers are unnecessary.
Choosing Complementary Batching Patterns: Distinguish when to use system-level batching versus the Batch API, including how both can serve different workloads in one system.
Why it matters: Choosing by urgency lets a system handle both interactive traffic and large background jobs without treating them as the same problem.

Key terms: system-level batching, queue, concurrency, traffic spikes, OpenAI Batch API, .jsonl, custom_id, batch job, Files API, real-time user queries

9.2.3 Caching for efficiency

This section explains how exact-match and semantic caching reduce repeated LLM calls while using similarity checks and cache invalidation to protect answer quality.

Exact and semantic cache matching: Distinguish exact string matching from semantic caching, and explain why reusing responses to repeated or rephrased queries reduces latency and cost.
Why it matters: Choosing the right kind of matching determines how many repeated questions can avoid another model call.
Semantic cache request flow: Trace how a semantic cache stores a query and response, looks up a new query, and decides whether to return a stored answer or call the LLM.
Why it matters: The lookup flow shows precisely when the system can skip generation and when it must create a fresh answer.
Similarity thresholds and fallback: Explain why the closest cached query may still be a poor match and how a similarity threshold and LLM fallback prevent an irrelevant cached answer.
Why it matters: A fast cache hit is harmful if it returns an answer to a different question.
FAISS and GPTCache tradeoff: Distinguish a manually managed FAISS-based cache from GPTCache in terms of setup, control, and maintenance.
Why it matters: The choice trades implementation effort against the ability to tailor cache behavior.
Stale answer prevention strategies: Explain why cached answers can become stale and describe ways to keep responses current.
Why it matters: A cache must save calls without continuing to serve answers after their source information changes.

Key terms: exact string matching, semantic caching, vector embeddings, embedding model, vector database, similarity search, cached response, similarity threshold, poor matches, LLM fallback

9.2.4 Multimodel fallback: Matching each query to the right model

This section explains how multimodel fallback routes queries by difficulty to balance answer quality, cost, and latency.

Tiered model routing: Explain why a router sends routine queries to a cheaper, faster model and escalates complex or specialized queries to a more capable one.
Why it matters: Matching model capability to the request preserves stronger answers where needed without paying for them on every query.
Query complexity routing methods: Describe how heuristics, a trained classifier, or a smaller model using function calling can decide where to route a query, including the advantage of the smaller-model approach over keyword matching.
Why it matters: The routing method affects both the cost of making a decision and how well the system recognizes unfamiliar difficult queries.
Two-pass quality escalation: Trace a two-pass fallback from a cheaper model’s first answer through a quality check to escalation, and explain its cost-versus-latency tradeoff.
Why it matters: A quality check can catch difficult requests that a simple length or keyword rule misses.
Routing errors and upkeep: Explain how misclassifying complex queries affects answer quality and why routing logic must be monitored and updated as user queries change.
Why it matters: Cost savings are not worthwhile if the router quietly sends difficult requests to a model that cannot answer them well.

Key terms: multimodel fallback, router, classifier, query complexity, heuristics, machine-learning classifier, function calling, decision layer, two-pass approach, confidence or quality check

9.2.5 Project: Building an e-commerce LLM service with batching, caching, and model fallback

Builds an e-commerce LLM service that uses queued batching, semantic caching, model routing, and optional streaming to handle busy periods efficiently.

Queued system-level batching: Trace how requests enter a shared queue, are collected into batches, and are dispatched concurrently to handle traffic surges.
Why it matters: Queuing and batching help the chatbot remain responsive when many customers arrive at once.
Near-duplicate answer reuse: Explain how the service uses GPTCache to answer repeated or sufficiently similar questions without another real model call.
Why it matters: Reusing answers saves model calls when customers ask the same question in different words.
Complexity-based model routing: Describe how the service chooses between its everyday and advanced models for each request, and why.
Why it matters: Routing avoids paying advanced-model rates for simple customer questions.
Optional streamed responses: Describe how a request selects streaming instead of a single response and what streaming changes for the user.
Why it matters: Streaming improves the perceived responsiveness of chatlike interactions without requiring every answer to use it.

Key terms: request_queue, enqueue_request, batch_size, system-level batching, GPTCache, semantic-level caching, ONNX embedding model, similarity threshold, detect_complexity, choose_model

9.2.6 Further improvements

The section outlines ways to refine a running e-commerce LLM service for cost, speed, routing accuracy, and freshness.

Usage and cost visibility: Explain what usage data to track in the e-commerce service and how it helps refine the system.
Why it matters: You need usage evidence to tell whether an optimization is saving resources or behaving unexpectedly.
Meaning-preserving prompt compression: Explain how prompt compression reduces latency and cost, and describe the role of meta prompting.
Why it matters: A shorter prompt can lower the cost and delay of repeated requests without changing what the request asks.
More selective model fallback: Describe two ways to improve which model handles a query: a two-pass confidence check and a better complexity classifier.
Why it matters: Better fallback avoids expensive-model calls for routine requests without leaving difficult ones with the cheaper model.
Freshness of cached answers: Explain how versioning or timestamps can prevent cached answers from becoming misleading when store information changes.
Why it matters: Reusing an answer saves work only if the underlying store information is still current.
Specialized models within existing patterns: Explain when a specialized model might help and how it fits alongside the service’s existing optimizations.
Why it matters: Specialization is another optimization option, not a replacement for the service’s caching and traffic-management patterns.
Parallel requests within batches: Describe the proposed change to batch processing and its tradeoff during larger traffic surges.
Why it matters: Handling a larger surge may require trading sequential execution for more concurrent model calls.

Key terms: tokens, gpt-5.4, gpt-5.4-mini, GPTCache, cached answers, prompt compression, token use, intent, meta prompting, two-pass fallback

9.3 Evaluating agent performance

This section explains why evaluating agents requires looking beyond standalone LLM metrics to assess both outcomes and the steps that produce them.

Agent evaluation criteria: Distinguish typical standalone LLM evaluation from agent evaluation. Explain that an agent must be assessed not just for output correctness but also for its actions, tool interactions, decision-making efficiency, and execution path.
Why it matters: An agent can produce a plausible answer while still taking an inefficient or wrong path to get there.
Pipeline and stepwise assessment: Explain why agent evaluation needs both an end-to-end assessment of the reasoning and execution pipeline and step-by-step analysis.
Why it matters: Looking at both the whole run and its individual steps makes failures easier to detect.

Key terms: accuracy, hallucination rate, latency, token efficiency, decision-making process, execution path, end-to-end assessments, reasoning and execution pipeline, granular step-by-step analysis, failure points

9.3.1 Core metrics for evaluating LLMs

The section explains why evaluating an agent requires looking beyond the quality of its final text output.

Limits of output-only evaluation: Explain why output quality metrics also apply to agents but cannot, on their own, reveal failures in an agent’s multistep work.
Why it matters: Judging only the final answer can miss problems in how an agent arrived at it.

Key terms: output quality metrics, agents, intent, tools, intermediate reasoning, tool call

9.3.2 Evaluating agents: Beyond traditional LLM metrics

This section explains how to evaluate an agent’s completed task and inspect its actions and execution steps for problems that a final answer alone may hide.

End-to-end agent evaluation: Explain why evaluating an agent requires more than judging its final text, and describe what a start-to-finish task test checks.
Why it matters: A convincing reply does not establish that the agent actually completed the user’s task.
Hidden workflow errors: Explain why an apparently correct final output may still require stage-by-stage inspection, and name the stages this can help diagnose.
Why it matters: Stage-by-stage inspection helps locate a failure instead of treating every bad workflow as a bad final answer.
Tool-call verification: Describe how to check whether an agent used the tools appropriate to a task.
Why it matters: Checking tool calls tests what the agent actually did, not merely what it claimed to do.
Execution trajectory comparison: Describe how comparing an execution trace with an expected trajectory helps evaluate and debug an agent’s workflow.
Why it matters: A trace comparison shows where an agent’s process diverged, making complex workflow failures easier to debug.

Key terms: end-to-end testing, intent classification, actions, final response, internal workflow, parameter extraction, tool calls, LangSmith, expected tool calls, trajectory

9.4 Summary

Where this is used later

Continuous quality monitoring with automated testing (chapter 10) · Identifying and measuring problems in LLM outputs
The six principles of reliable AI (chapter 11) · evaluating LLM output failures
Output-quality monitoring (chapter 10) · Output-quality monitoring can use measured grounding defects and their trends.
Anthropic’s constitutional AI: LLM-as-a-judge at training scale (chapter 11) · the earlier use of an LLM judge to evaluate model outputs
Building LLM-native monitoring systems (chapter 10) · Production monitoring builds on the need to observe problems continuously and at scale.
Setting up alerts that help (chapter 10) · Threshold-based alerts build on detecting a monitored problem that needs attention.
Logging what matters (chapter 10) · Introduces response streaming as a way to present answers incrementally.
The hybrid solution: Best of both worlds (chapter 10) · Matching queries to models through multimodel fallback

Concepts explained in this chapter

semantic cachingA caching technique that reuses results for queries with similar meanings, rather than requiring exact textual matches.
batchingTemporarily grouping incoming application requests to manage concurrency and reduce processing overhead.
multimodel fallbackA model-routing strategy that sends simpler queries to a cheaper model and escalates harder or inadequately answered queries to a more capable model.
token streamingThe incremental delivery of generated text as tokens become available, rather than waiting for a complete response.
routingThe technique of directing different queries or tasks to selected models according to their requirements or cost.
GPTCacheGPTCache is a library that intercepts requests to GPT-based endpoints and caches exact or semantically similar queries.
cachingStoring results so they can be reused instead of recomputed or retrieved again.
two-pass routingA method that evaluates a task with one model, then escalates to another model if the first result is inadequate.
grounding defect rateGrounding Defect Rate is a metric for quantifying the rate of unsupported claims in generated responses.
adversarial test setA collection of challenging or ambiguous inquiries designed to expose likely model errors such as hallucinations.
grounding datasetA collection of authoritative information or evidence used to support or verify system responses.
FActScoreFActScore is a metric that evaluates the factual accuracy of generated text by decomposing it into atomic facts and checking each fact.
stress testStress testing evaluates a system under unusually high loads or demanding conditions to identify weaknesses and failure modes.
monitoringThe ongoing observation of a deployed AI system's behavior, performance, and quality.
evaluatorAn evaluation technique in which a large language model assesses the quality of another model's output.
end-to-end testingTesting that evaluates a system by simulating a complete task from its initial input through its final outcome.
hallucinationA hallucination is a model-generated output that presents fabricated or unsupported information as if it were true.
red-teamingThe practice of deliberately probing a system for weaknesses, unsafe behavior, or failure modes.
exact string matchingA matching technique that considers two strings a match only when their text is identical.
versioningA technique for associating entries with versions so entries from an outdated version can be invalidated.
lexical searchA retrieval method that searches using sparse representations of content, such as term-based vectors.
semantic similarityA measure of how closely the meanings of two pieces of content correspond.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.