Chapter 8: Evaluation and observability
The introduction positions evaluation and observability as assurance disciplines that help diagnose context problems and assess whether LLM-based systems behave as intended.
After this chapter you will be able to
- Evaluation and observability roles. Explain why evaluation and observability support the context engineering stack: they provide feedback, automated graders, and telemetry to diagnose context degradation, assess retrieval quality, and check whether multi-step agent behavior stays aligned with user intent.
- Context failure definition. Explain that a context failure occurs when information supplied to an LLM-based system does not support the task, user intent, or expected agent behavior, making a correct answer less likely.
- Diagnostic failure taxonomy. Describe how the taxonomy groups context failures by main cause and explain why its categories can overlap in one incident.
- Context rot and mitigation. Explain how accumulated context can lose usefulness during a long conversation or multi-step agent run, how this differs from mere staleness, and what the section recommends doing about it.
- Context staleness and freshness controls. Explain when context becomes stale, give an example, and describe the controls the section names for managing freshness.
- Context drift and prevention. Explain how a workflow can gradually produce an answer to a related but unrequested task, and describe how to keep it aligned with the original objective.
- Context poisoning and source isolation. Explain how adversarial or untrusted content in retrieved documents, web pages, emails, or tool outputs can try to override instructions or trigger unsafe behavior. Describe source isolation as keeping that content as data rather than instructions, while enforcing instruction hierarchy and tracking provenance.
- Context pollution and mitigation. Explain what context pollution is, where it commonly comes from, how it affects answers, and what can reduce it.
- Context distraction and salience control. Explain how irrelevant but salient context can divert a model from task-critical evidence, and describe the mitigations named in this section.
- Context confusion and mitigation. Explain how context confusion occurs and describe the safeguards named in this section.
- Context clash and response. Explain what makes context items clash, give an example from the section, and describe how the system should respond.
- Context overflow and mitigation. Explain when context overflow occurs, why a token limit is not the only possible constraint, how overflow can lose important information, and the mitigations named here.
- Evidence-based context evaluation. Explain why systematic evaluation is preferable to checking a few prompts and deciding that the responses look good. Describe how explicit criteria provide a repeatable way to judge whether a context strategy improves system behavior.
- Targeted context survival probes. Describe how small, targeted evaluation probes check whether critical information survives context selection, writing, compression, or isolation. Explain why these checks complement broad quality rubrics and can reveal failures that generic similarity scores miss.
- Agent Eval Scope. Explain why an agent eval must inspect the run and its effects, not just the final response.
- Eval Components and Trials. Describe how a task, grader, transcript, outcome, harness, and suite fit into an agent evaluation, and why a task may need multiple trials.
- Grader Type Tradeoffs. Distinguish code-based, model-based, and human graders by what they handle well and their principal limitations.
- Capability and Regression Suites. Distinguish capability evaluations from regression evaluations and describe when cases move between them.
- Comparable performance metrics. Explain why evaluation metrics must measure specific performance dimensions consistently across runs and how teams use them to set passing thresholds and track changes.
- Operational performance envelopes. Explain why run duration, token use, and model and tool calls should be evaluated alongside answer quality, including what happens when a run exceeds its expected operating envelope.
- Scoring method tradeoffs. Distinguish statistical, model-based, and LLM-based ways to score outputs, including why semantic judgments may require more than string overlap.
- RAG evaluation triad. Explain the three RAG evaluation questions: whether retrieval found relevant context, whether the answer is grounded in that context, and whether it addresses the user's intent.
- Context-Aware Test Case Coverage. Explain what an evaluation dataset measures and why its cases should test both expected uses of context and situations where the system must reject or question misleading or insufficient context.
- Dataset Quality Criteria. Describe how representative, stratified, reviewed, and versioned cases make an evaluation dataset useful.
- Repeatable benchmark setup. Explain how a benchmark makes performance comparisons repeatable across implementations or system versions.
- Component versus end-to-end benchmarks. Distinguish benchmarks of individual stack components from benchmarks of a complete context-aware application or agent.
- End-to-end benchmark cases. Describe what an end-to-end context benchmark item supplies and what kinds of context-dependent behavior it can test.
- Representative and protected suites. Justify using varied difficulty levels, reporting results by stratum and overall, and keeping evaluation sets versioned and separate from tuning material.
- LLM-as-judge evaluation pattern. Explain how a separate LLM evaluates a system’s candidate response using the query, output, and relevant context or reference material, and returns a quality judgment rather than a user-facing answer.
- Direct and pairwise judging. Distinguish direct scoring of one output against a rubric from pairwise comparison of two outputs, including the kinds of criteria each fits.
- Human-label calibration process. Describe how to calibrate a judge against labeled examples and improve its rubric when disagreements cluster.
- Release-gate safeguards. Explain the checks needed before using judge scores to gate a release and when people should remain involved.
- Evaluation and observability roles. Distinguish evaluation’s role in identifying whether a context-aware system meets its quality bar from observability’s role in explaining why it behaved as it did. Explain how observability connects a visible symptom to the context and deployment state that shaped the answer.
- Monitoring and observability roles. Distinguish monitoring’s detection of predefined signals from observability’s use of runtime evidence to explain unexpected system behavior.
- Three telemetry data pillars. Describe logs, metrics, and traces as distinct forms of structured runtime telemetry and what each reveals.
- Limits of passive telemetry. Explain why traditional telemetry alone can miss context-related AI failures and why retrospective, passive collection is insufficient for an agent needing immediate feedback.
- Context-rich telemetry feedback. Explain how active telemetry enriches runtime events with context and uses them to drive system decisions and evaluation, while recording why the system changed its behavior.
- Incident trace diagnosis. Trace how the support-assistant incident evidence separates a wrong answer into a stale fallback index and an orchestration race.
- Context graph structure. Describe a context graph as nodes for the artifacts and runtime events shaping an AI interaction, connected by edges that record their relationships.
- Relational failure diagnosis. Explain why a context graph can reveal failures that successful logs, traces, or a quality score may miss, using the pricing incident as an example.
- Stable identifiers across telemetry. Justify assigning stable identifiers to prompts, documents, indexes, tools, and deployments and carrying them unchanged through telemetry.
- Standardized telemetry foundation. Explain how OpenTelemetry’s GenAI semantic conventions make context-engineering traces and metrics portable to general monitoring backends.
- AI-specific payload tracing. Distinguish what general monitoring backends track from the LLM-specific context captured by dedicated AI observability platforms.
- Trace correlation and protection. Describe what a robust observability stack should connect to execution traces and what protections it must apply before exporting them.
- Multi-metric DeepEval testing. Explain when DeepEval is useful for evaluating an LLM interaction and why keeping several quality checks with the test case helps maintain regression suites.
- Prompt comparison with Promptfoo. Explain how Promptfoo can compare prompt variants over the same test set, using both a model-graded rubric and a deterministic assertion in the section’s support-prompt example.
- Ragas evaluation signals. Describe how the example uses Ragas to evaluate a simple RAG pipeline on a small benchmark dataset, producing scores for faithfulness, answer relevancy, context precision, and context recall.
- Judge score calibration. Explain why the Ragas scores should be treated as decision-support data and calibrated against deterministic checks or human-labeled examples before serving as release gates.
- Langfuse agent execution traces. Explain how a Langfuse callback handler captures a tool-using agent’s execution and why the resulting trace is useful. Describe the kinds of information the trace may contain without treating every span as guaranteed.
- Traceable context assembly. Explain how the LangSmith example makes context assembly observable and why its trace metadata is useful.
- Evaluation and observability feedback loop. Describe how evaluation and observability work together to improve a production system.
What it builds on
- The context engineering stack (chapter 1) · Introduces the context engineering stack and its functional layers.
- Context selection (chapter 7) · Assumes that context can be selected for what a task needs.
- Context management (chapter 7) · The idea that context is actively managed through operations such as selection, writing, compression, and isolation.
- AI agents (chapter 1) · Agents can take actions beyond producing a response.
- Retrieval-augmented generation (chapter 3) · RAG has retrieval and generation components to assess separately.
Section by section
Introduction
The introduction positions evaluation and observability as assurance disciplines that help diagnose context problems and assess whether LLM-based systems behave as intended.
- Evaluation and observability roles: Explain why evaluation and observability support the context engineering stack: they provide feedback, automated graders, and telemetry to diagnose context degradation, assess retrieval quality, and check whether multi-step agent behavior stays aligned with user intent.
Why it matters: These disciplines help you tell when the context supplied to a system or its resulting behavior is going wrong.
Key terms: context engineering stack, cross-cutting assurance disciplines, evaluation, observability, feedback loops, automated graders
8.1 A taxonomy of context failures
Defines context failures and introduces a diagnostic taxonomy whose categories can overlap.
- Context failure definition: Explain that a context failure occurs when information supplied to an LLM-based system does not support the task, user intent, or expected agent behavior, making a correct answer less likely.
Why it matters: It directs attention to the supplied information, not just the model, when diagnosing a bad answer. - Diagnostic failure taxonomy: Describe how the taxonomy groups context failures by main cause and explain why its categories can overlap in one incident.
Why it matters: A single incident may need more than one context-related cause identified.
Key terms: context failures, LLM-based system, taxonomy of context failures, degradation, misalignment, contamination, interference, capacity limits
8.1.1 Context rot
Defines context rot, distinguishes it from staleness, and names ways to limit it.
- Context rot and mitigation: Explain how accumulated context can lose usefulness during a long conversation or multi-step agent run, how this differs from mere staleness, and what the section recommends doing about it.
Why it matters: An agent can be led astray by context that remains in its prompt after it has stopped being useful.
Key terms: context rot, context staleness, context pruning, explicit task state
8.1.2 Context staleness
Defines context staleness and identifies ways to keep changing information current.
- Context staleness and freshness controls: Explain when context becomes stale, give an example, and describe the controls the section names for managing freshness.
Why it matters: A piece of context can have been correct when captured but mislead the system after circumstances change.
Key terms: context staleness, timestamps, validity windows, invalidation rules, live lookups
8.1.3 Context drift
Defines context drift, explains how it arises in long workflows, and identifies ways to keep work aligned with the original goal.
- Context drift and prevention: Explain how a workflow can gradually produce an answer to a related but unrequested task, and describe how to keep it aligned with the original objective.
Why it matters: A workflow can appear sound step by step yet still miss what the user asked for.
Key terms: context drift, original goal, acceptance criteria, stable task contract, goal restatement, milestone checks
8.1.4 Context poisoning
Defines context poisoning and explains source isolation as the key defense against untrusted content that tries to act as instructions.
- Context poisoning and source isolation: Explain how adversarial or untrusted content in retrieved documents, web pages, emails, or tool outputs can try to override instructions or trigger unsafe behavior. Describe source isolation as keeping that content as data rather than instructions, while enforcing instruction hierarchy and tracking provenance.
Why it matters: It helps you recognize when material the system should analyze is trying to direct its behavior instead.
Key terms: context poisoning, source isolation, instruction hierarchy, provenance
8.1.5 Context pollution
Defines context pollution, identifies common sources and effects, and names ways to reduce it.
- Context pollution and mitigation: Explain what context pollution is, where it commonly comes from, how it affects answers, and what can reduce it.
Why it matters: Recognizing pollution helps you improve the context rather than treating a weak answer as solely a model problem.
Key terms: context pollution, retrieval ranking, deduplication, chunk quality checks, context inclusion
8.1.6 Context distraction
Defines context distraction, shows how salient but irrelevant material can derail an answer, and names ways to reduce it.
- Context distraction and salience control: Explain how irrelevant but salient context can divert a model from task-critical evidence, and describe the mitigations named in this section.
Why it matters: A model can miss the main requirement even when the necessary evidence is present.
Key terms: context distraction, task-critical evidence, salience control
8.1.7 Context confusion
Defines context confusion as mixing up similar context items and names ways to keep them distinct.
- Context confusion and mitigation: Explain how context confusion occurs and describe the safeguards named in this section.
Why it matters: Keeping context items distinct helps prevent the system from acting on the wrong user, file, tool, or requirement.
Key terms: context confusion, stable identifiers, explicit role labels, normalized constraints, structured state
8.1.8 Context clash
Defines context clash as incompatible context items and states how a system should handle the conflict.
- Context clash and response: Explain what makes context items clash, give an example from the section, and describe how the system should respond.
Why it matters: Recognizing a clash helps prevent a system from hiding incompatible instructions or information.
Key terms: context clash, instruction precedence, unresolved contradictions
8.1.9 Context overflow
Defines context overflow, explains how it can erase decisive information, and links context failure types to evaluation signals.
- Context overflow and mitigation: Explain when context overflow occurs, why a token limit is not the only possible constraint, how overflow can lose important information, and the mitigations named here.
Why it matters: An agent can lose instructions it needs even when the problem is not simply a full token window. - Failure specific evaluation signals: Describe how the failure taxonomy suggests different checks for context quality, including the checks associated with overflow.
Why it matters: Matching a check to the failure makes it more likely to reveal the particular context problem.
Key terms: context overflow, budget-aware context assembly, retrieval on demand, structured summaries, memory segmentation, evaluation signal, longitudinal checks, freshness checks, task-alignment checks, provenance
8.2 Evaluation
This section explains why context engineering needs systematic evaluation and why targeted probes are needed to catch information lost during context management.
- Evidence-based context evaluation: Explain why systematic evaluation is preferable to checking a few prompts and deciding that the responses look good. Describe how explicit criteria provide a repeatable way to judge whether a context strategy improves system behavior.
Why it matters: It gives engineers evidence for changing a context strategy rather than relying on a few promising responses. - Targeted context survival probes: Describe how small, targeted evaluation probes check whether critical information survives context selection, writing, compression, or isolation. Explain why these checks complement broad quality rubrics and can reveal failures that generic similarity scores miss.
Why it matters: A response can appear broadly good while losing a fact, instruction, or next step needed to continue the task.
Key terms: systematic evaluation, informal validation, explicit criteria, probe, compression, degradation, critical information
8.2.1 Evals
This section explains how to construct and use agent evaluations, choose graders, distinguish capability from regression tests, and keep evaluation suites informative as agents improve.
- Agent Eval Scope: Explain why an agent eval must inspect the run and its effects, not just the final response.
Why it matters: An agent can produce a plausible answer while taking the wrong actions or leaving the environment in the wrong state. - Eval Components and Trials: Describe how a task, grader, transcript, outcome, harness, and suite fit into an agent evaluation, and why a task may need multiple trials.
Why it matters: Separating the test, its recorded run, and its result makes an agent’s performance possible to inspect and grade. - Grader Type Tradeoffs: Distinguish code-based, model-based, and human graders by what they handle well and their principal limitations.
Why it matters: Choosing complementary graders helps balance reliable checks, nuanced judgment, and review cost. - Capability and Regression Suites: Distinguish capability evaluations from regression evaluations and describe when cases move between them.
Why it matters: Improvements need tests that reveal new progress without letting previously working behavior break. - Evaluation Driven Development and Saturation: Describe evaluation-driven development and explain how to respond when a suite saturates.
Why it matters: A suite should guide development and remain able to show meaningful improvement. - Individual Evals and Aggregate Signals: Explain why passing individual evals does not by itself establish strong aggregate behavior.
Why it matters: Passing one case can conceal recurring quality or efficiency problems.
Key terms: AI evals, trajectory, final environment state, task, grader, transcript, outcome, evaluation harness, agent harness, code-based graders
8.2.2 Evaluation metrics
This section explains what evaluation metrics measure, how scores are calculated, and which aspects of RAG systems and agent workflows need evaluation.
- Comparable performance metrics: Explain why evaluation metrics must measure specific performance dimensions consistently across runs and how teams use them to set passing thresholds and track changes.
Why it matters: A score is useful when it helps you compare runs and decide whether a change is acceptable. - Operational performance envelopes: Explain why run duration, token use, and model and tool calls should be evaluated alongside answer quality, including what happens when a run exceeds its expected operating envelope.
Why it matters: A correct result is not enough if the workflow becomes unexpectedly expensive or inefficient. - Scoring method tradeoffs: Distinguish statistical, model-based, and LLM-based ways to score outputs, including why semantic judgments may require more than string overlap.
Why it matters: The scoring method affects which qualities a metric can reliably detect. - Structured LLM scoring patterns: Describe how G-Eval, decision-tree metrics, and question-answer generation structure an LLM's judgment rather than relying on one unstructured rating.
Why it matters: Structured judgments make it clearer what a score is based on. - RAG evaluation triad: Explain the three RAG evaluation questions: whether retrieval found relevant context, whether the answer is grounded in that context, and whether it addresses the user's intent.
Why it matters: The triad helps distinguish a retrieval problem from an ungrounded or off-target answer. - Agent workflow test cases: Explain why evaluating an agent requires tracing its execution flow to construct test cases across components.
Why it matters: A trace makes the workflow's component-level behavior available for evaluation.
Key terms: evaluation metrics, minimum passing thresholds, operational metrics, operating envelope, run duration, token consumption, model and tool calls, statistical scorers, model-based scorers, LLM-based scoring
8.2.3 Evaluation datasets
Explains how to build context-aware evaluation datasets and choose test-case sources while accounting for their limitations.
- Context-Aware Test Case Coverage: Explain what an evaluation dataset measures and why its cases should test both expected uses of context and situations where the system must reject or question misleading or insufficient context.
Why it matters: Testing only successful answers misses important failures in how a system handles context. - Dataset Quality Criteria: Describe how representative, stratified, reviewed, and versioned cases make an evaluation dataset useful.
Why it matters: These practices help make scores relevant and prevent an overall score from concealing weaknesses. - Test Case Source Tradeoffs: Distinguish public, human-authored, synthetic, and production-derived dataset sources by what each contributes and where it falls short or needs safeguards.
Why it matters: The source of test cases affects both their relevance and the checks needed before using them.
Key terms: evaluation dataset, test cases, negative cases, Representative, Stratified, Reviewed, Versioned, public datasets, human-authored datasets, synthetic datasets
8.2.4 Benchmarks
This section explains what benchmarks measure in context engineering, how to design end-to-end suites, and what benchmarking alone cannot score.
- Repeatable benchmark setup: Explain how a benchmark makes performance comparisons repeatable across implementations or system versions.
Why it matters: Without a stable setup, a change in results may reflect a change in the test rather than the system. - Component versus end-to-end benchmarks: Distinguish benchmarks of individual stack components from benchmarks of a complete context-aware application or agent.
Why it matters: A strong model or retriever score cannot substitute for testing the assembled application. - End-to-end benchmark cases: Describe what an end-to-end context benchmark item supplies and what kinds of context-dependent behavior it can test.
Why it matters: The expected handling of context matters as well as the final answer. - Representative and protected suites: Justify using varied difficulty levels, reporting results by stratum and overall, and keeping evaluation sets versioned and separate from tuning material.
Why it matters: A trustworthy benchmark must expose difficult-case failures without benefiting from leaked examples. - Benchmark scoring limits: Explain why a repeatable benchmark does not by itself determine how to score nuanced qualities such as faithfulness or relevance.
Why it matters: Defining consistent test cases and deciding how to judge their responses are separate tasks.
Key terms: benchmark, inputs, conditions, success criteria, component benchmarks, end-to-end context benchmarks, MMLU, HELM, MTEB, BEIR
8.2.5 LLM-as-judge
This section explains how LLM-as-judge produces scalable evaluation scores, when to use its two common patterns, and why its results need calibration, checks, and human oversight.
- LLM-as-judge evaluation pattern: Explain how a separate LLM evaluates a system’s candidate response using the query, output, and relevant context or reference material, and returns a quality judgment rather than a user-facing answer.
Why it matters: This pattern makes qualities that are hard to score with simple heuristics assessable across many test cases. - Direct and pairwise judging: Distinguish direct scoring of one output against a rubric from pairwise comparison of two outputs, including the kinds of criteria each fits.
Why it matters: Choosing the judging pattern to match the criterion makes the resulting comparison more meaningful. - Judge biases and limits: Describe position, verbosity, and self-enhancement bias, and explain why a judge score is not fully objective.
Why it matters: These biases explain why an automated score needs validation before it drives decisions. - Human-label calibration process: Describe how to calibrate a judge against labeled examples and improve its rubric when disagreements cluster.
Why it matters: Calibration tests whether the judge’s scores agree with the judgments the evaluation is meant to reflect. - Release-gate safeguards: Explain the checks needed before using judge scores to gate a release and when people should remain involved.
Why it matters: A scalable judge should not become an unstable or unchecked release decision-maker. - Evaluation diagnostic gap: Distinguish a failing evaluation score from evidence about what caused the failure.
Why it matters: Knowing that an answer failed is different from having the evidence needed to diagnose it.
Key terms: LLM-as-judge, structured evaluation prompt, numerical score, justification, direct scoring, rubric, pairwise comparison, position bias, verbosity bias, self-enhancement bias
8.3 Observability
This section distinguishes evaluation from observability and explains how they work together to diagnose a context-aware system’s behavior.
- Evaluation and observability roles: Distinguish evaluation’s role in identifying whether a context-aware system meets its quality bar from observability’s role in explaining why it behaved as it did. Explain how observability connects a visible symptom to the context and deployment state that shaped the answer.
Why it matters: The distinction helps a team move from noticing a poor answer to investigating what shaped it.
Key terms: evaluation, observability, context, deployment state
8.3.1 Traditional observability foundations
Defines traditional observability and its telemetry pillars, then explains why context-aware AI systems need more than passive infrastructure telemetry.
- Monitoring and observability roles: Distinguish monitoring’s detection of predefined signals from observability’s use of runtime evidence to explain unexpected system behavior.
Why it matters: Detecting a problem and explaining its cause require different kinds of evidence. - Three telemetry data pillars: Describe logs, metrics, and traces as distinct forms of structured runtime telemetry and what each reveals.
Why it matters: Each telemetry type answers a different question about what a system did. - Limits of passive telemetry: Explain why traditional telemetry alone can miss context-related AI failures and why retrospective, passive collection is insufficient for an agent needing immediate feedback.
Why it matters: Healthy infrastructure signals can coexist with a context failure that an agent needs to address promptly.
Key terms: monitoring, observability, telemetry, structured telemetry, logs, metrics, traces, spans, context-aware AI systems, infrastructure traces
8.3.2 Active telemetry
This section explains how active telemetry turns context-rich runtime events into decisions and evaluation signals, then uses a support-assistant incident to show how traces reveal the causes of a wrong answer.
- Context-rich telemetry feedback: Explain how active telemetry enriches runtime events with context and uses them to drive system decisions and evaluation, while recording why the system changed its behavior.
Why it matters: A trace can support action during a run, not just explain a failure afterward. - Adaptive and readable traces: Describe how telemetry capture changes with runtime conditions and why its labels and structure matter to people inspecting decisions.
Why it matters: Adaptive capture preserves detail where it is most useful without tracing every routine request equally. - Incident trace diagnosis: Trace how the support-assistant incident evidence separates a wrong answer into a stale fallback index and an orchestration race.
Why it matters: A plausible-looking answer can hide multiple defects that require different fixes and eval cases.
Key terms: active telemetry, semantic metadata, bidirectional feedback, dynamic sampling and enrichment, human-readable semantics, model-graded evaluations, model span, retrieval span, session logs, stale fallback index
8.3.3 Context graphs
This section explains how context graphs reveal relationships behind AI failures and support incident review without requiring a graph database.
- Context graph structure: Describe a context graph as nodes for the artifacts and runtime events shaping an AI interaction, connected by edges that record their relationships.
Why it matters: The graph makes it possible to inspect how pieces of context were connected, not just whether they appeared in a run. - Relational failure diagnosis: Explain why a context graph can reveal failures that successful logs, traces, or a quality score may miss, using the pricing incident as an example.
Why it matters: It helps distinguish an incorrect answer caused by bad context assembly from one caused by the model ignoring good evidence. - Incident review graph queries: Describe how a simple table of relationship records can support incident review, including questions about affected sessions, shared failure causes, change impact, and decision provenance.
Why it matters: These queries turn scattered complaints and run records into actionable incident evidence. - Stable identifiers across telemetry: Justify assigning stable identifiers to prompts, documents, indexes, tools, and deployments and carrying them unchanged through telemetry.
Why it matters: Without stable IDs, the relationships needed to investigate an incident cannot be reliably followed.
Key terms: context graph, nodes, edges, relational failures, superseded_by, orchestration race, groundedness score, table of triples, blast radius, failure grouping
8.3.4 Observability tools
This section explains how standardized telemetry and AI-specific tracing tools work together, and what a robust observability stack must correlate and protect.
- Standardized telemetry foundation: Explain how OpenTelemetry’s GenAI semantic conventions make context-engineering traces and metrics portable to general monitoring backends.
Why it matters: Standardized telemetry lets an observability stack use general monitoring backends without a custom adapter for each vendor. - AI-specific payload tracing: Distinguish what general monitoring backends track from the LLM-specific context captured by dedicated AI observability platforms.
Why it matters: Infrastructure metrics alone do not show the context an LLM used during an execution. - Trace correlation and protection: Describe what a robust observability stack should connect to execution traces and what protections it must apply before exporting them.
Why it matters: Linked trace context supports review, while export safeguards protect data that AI traces may contain.
Key terms: OpenTelemetry, GenAI semantic conventions, traces, metrics, general backends, dedicated AI observability platforms, LLM-specific payload context, execution traces, prompt versions, retrieval indexes
8.4 Hands-on
Introduces hands-on evaluation and observability examples and points readers to the companion repository for complete scripts and setup instructions.
8.4.1 Metrics with DeepEval
This section presents DeepEval as a way to score one LLM interaction with multiple metrics while keeping evaluation logic close to the test case.
- Multi-metric DeepEval testing: Explain when DeepEval is useful for evaluating an LLM interaction and why keeping several quality checks with the test case helps maintain regression suites.
Why it matters: Multiple checks let you evaluate different aspects of a reply without separating the checks from the test they assess.
Key terms: DeepEval, AnswerRelevancyMetric, FaithfulnessMetric, GEval, regression suites
8.4.2 Evals with Promptfoo
This section presents Promptfoo as a config-driven tool for comparing prompts and checking LLM application behavior.
- Prompt comparison with Promptfoo: Explain how Promptfoo can compare prompt variants over the same test set, using both a model-graded rubric and a deterministic assertion in the section’s support-prompt example.
Why it matters: A shared test set makes prompt comparisons more meaningful. - Config-driven evaluation benefits: Describe why the section presents Promptfoo as useful for prompt iteration and regression checks, including the visibility of its config-driven setup.
Why it matters: A visible evaluation setup is easier to review as prompts change.
Key terms: Promptfoo, prompt variants, test set, model-graded rubric, deterministic assertion, prompt iteration, regression checks, config-driven, audit
8.4.3 LLM-as-judge with Ragas
The section uses a small RAG evaluation example to show what Ragas judge scores can signal and why they need calibration before guiding releases.
- Ragas evaluation signals: Describe how the example uses Ragas to evaluate a simple RAG pipeline on a small benchmark dataset, producing scores for faithfulness, answer relevancy, context precision, and context recall.
Why it matters: These scores help assess both the context a RAG pipeline retrieves and the answers it generates. - Judge score calibration: Explain why the Ragas scores should be treated as decision-support data and calibrated against deterministic checks or human-labeled examples before serving as release gates.
Why it matters: Calibration keeps useful judge scores from becoming unverified pass-or-fail criteria.
Key terms: LLM-as-judge, Ragas, faithfulness, answer relevancy, context precision, context recall, decision-support data, deterministic checks, human-labeled examples, release gates
8.4.4 Observability with Langfuse
This section shows how a Langfuse callback handler records a small agent’s execution for observability and debugging.
- Langfuse agent execution traces: Explain how a Langfuse callback handler captures a tool-using agent’s execution and why the resulting trace is useful. Describe the kinds of information the trace may contain without treating every span as guaranteed.
Why it matters: A trace helps you investigate context-related failures by showing what happened during an agent run.
Key terms: Langfuse, callback handler, trace, spans
8.4.5 Observability with LangSmith
This section uses LangSmith to show how tracing context assembly supports observability, then connects runtime traces to an evaluation-and-improvement feedback loop.
- Traceable context assembly: Explain how the LangSmith example makes context assembly observable and why its trace metadata is useful.
Why it matters: A trace lets you inspect the context supplied to a run rather than judging only its final output. - Evaluation and observability feedback loop: Describe how evaluation and observability work together to improve a production system.
Why it matters: This loop turns production problems into testable improvements rather than isolated incidents.
Key terms: LangSmith, context assembly, trace metadata, active telemetry, evaluation, observability, eval cases, traces, feedback loop
8.5 Summary
Where this is used later
- AI application frameworks (chapter 10) · The possibility of failures arising from the context supplied to an AI system.
- Context quality (chapter 12) · Context failure modes that can be investigated.
- Context selection (chapter 12) · The previously introduced term context rot.
- Governance (chapter 9) · Introduces evaluation, one activity that governance can set policies for.
- Evaluation (chapter 11) · The general role of evaluation in AI systems.
- RAGOps (chapter 9) · Retriever and generator evaluation metrics
- Human-in-the-loop (chapter 9) · The use of datasets in evaluation.
- Privacy (chapter 9) · The role of observability traces in recording application activity.
Concepts explained in this chapter
- evaluationEvaluation is the structured testing and measurement of an AI system's behavior and performance.
- LLM-as-a-judgeAn evaluator is a component that assesses outcomes against specified criteria or metrics.
- context graphA graph representation of the artifacts and runtime events that shape an AI system’s behavior during an interaction, task, or workflow.
- observabilityObservability is the ability to understand a system's internal behavior through signals such as logs, traces, and metrics.
- trajectoryA trace describes a request or operation’s flow across components, typically as a sequence of related spans.
- telemetryAutomatically collected runtime evidence about a software system's operation, including events, measurements, and request-flow data.
- evaluation-driven developmentA software engineering paradigm that makes evaluations the proactive driver of development.
- test suiteA test suite is a collection of software tests run to check whether a program behaves as expected.
- evaluation datasetAn evaluation set is a collection of test cases and expected or judged outcomes used to assess a system's performance.
- redactionThe removal or masking of sensitive information from data before it is shared, stored, exposed, or processed.
- context rotContext rot is degradation in the usefulness of accumulated context as a task evolves and its information loses relevance, structure, or priority.
- isolationSeparating context content or reasoning scopes to control information exposure.
- stale dataStale data is information that is outdated because it does not reflect events or changes after the data’s last update.
- resultThe final state of an environment at the end of an evaluation trial.
- taskAn A2A task is a stateful unit of work with an explicit lifecycle that can support long-running operations and multi-turn collaboration.
- monitoringMonitoring is the ongoing collection and examination of system signals and events to assess behavior and health.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
