liveBook · Manning

Chapter 8: Evaluation and observability

What you will learn in chapter 8 of Context Engineering

The introduction positions evaluation and observability as assurance disciplines that help diagnose context problems and assess whether LLM-based systems behave as intended.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction positions evaluation and observability as assurance disciplines that help diagnose context problems and assess whether LLM-based systems behave as intended.

Key terms: context engineering stack, cross-cutting assurance disciplines, evaluation, observability, feedback loops, automated graders

8.1 A taxonomy of context failures

Defines context failures and introduces a diagnostic taxonomy whose categories can overlap.

Key terms: context failures, LLM-based system, taxonomy of context failures, degradation, misalignment, contamination, interference, capacity limits

8.1.1 Context rot

Defines context rot, distinguishes it from staleness, and names ways to limit it.

Key terms: context rot, context staleness, context pruning, explicit task state

8.1.2 Context staleness

Defines context staleness and identifies ways to keep changing information current.

Key terms: context staleness, timestamps, validity windows, invalidation rules, live lookups

8.1.3 Context drift

Defines context drift, explains how it arises in long workflows, and identifies ways to keep work aligned with the original goal.

Key terms: context drift, original goal, acceptance criteria, stable task contract, goal restatement, milestone checks

8.1.4 Context poisoning

Defines context poisoning and explains source isolation as the key defense against untrusted content that tries to act as instructions.

Key terms: context poisoning, source isolation, instruction hierarchy, provenance

8.1.5 Context pollution

Defines context pollution, identifies common sources and effects, and names ways to reduce it.

Key terms: context pollution, retrieval ranking, deduplication, chunk quality checks, context inclusion

8.1.6 Context distraction

Defines context distraction, shows how salient but irrelevant material can derail an answer, and names ways to reduce it.

Key terms: context distraction, task-critical evidence, salience control

8.1.7 Context confusion

Defines context confusion as mixing up similar context items and names ways to keep them distinct.

Key terms: context confusion, stable identifiers, explicit role labels, normalized constraints, structured state

8.1.8 Context clash

Defines context clash as incompatible context items and states how a system should handle the conflict.

Key terms: context clash, instruction precedence, unresolved contradictions

8.1.9 Context overflow

Defines context overflow, explains how it can erase decisive information, and links context failure types to evaluation signals.

Key terms: context overflow, budget-aware context assembly, retrieval on demand, structured summaries, memory segmentation, evaluation signal, longitudinal checks, freshness checks, task-alignment checks, provenance

8.2 Evaluation

This section explains why context engineering needs systematic evaluation and why targeted probes are needed to catch information lost during context management.

Key terms: systematic evaluation, informal validation, explicit criteria, probe, compression, degradation, critical information

8.2.1 Evals

This section explains how to construct and use agent evaluations, choose graders, distinguish capability from regression tests, and keep evaluation suites informative as agents improve.

Key terms: AI evals, trajectory, final environment state, task, grader, transcript, outcome, evaluation harness, agent harness, code-based graders

8.2.2 Evaluation metrics

This section explains what evaluation metrics measure, how scores are calculated, and which aspects of RAG systems and agent workflows need evaluation.

Key terms: evaluation metrics, minimum passing thresholds, operational metrics, operating envelope, run duration, token consumption, model and tool calls, statistical scorers, model-based scorers, LLM-based scoring

8.2.3 Evaluation datasets

Explains how to build context-aware evaluation datasets and choose test-case sources while accounting for their limitations.

Key terms: evaluation dataset, test cases, negative cases, Representative, Stratified, Reviewed, Versioned, public datasets, human-authored datasets, synthetic datasets

8.2.4 Benchmarks

This section explains what benchmarks measure in context engineering, how to design end-to-end suites, and what benchmarking alone cannot score.

Key terms: benchmark, inputs, conditions, success criteria, component benchmarks, end-to-end context benchmarks, MMLU, HELM, MTEB, BEIR

8.2.5 LLM-as-judge

This section explains how LLM-as-judge produces scalable evaluation scores, when to use its two common patterns, and why its results need calibration, checks, and human oversight.

Key terms: LLM-as-judge, structured evaluation prompt, numerical score, justification, direct scoring, rubric, pairwise comparison, position bias, verbosity bias, self-enhancement bias

8.3 Observability

This section distinguishes evaluation from observability and explains how they work together to diagnose a context-aware system’s behavior.

Key terms: evaluation, observability, context, deployment state

8.3.1 Traditional observability foundations

Defines traditional observability and its telemetry pillars, then explains why context-aware AI systems need more than passive infrastructure telemetry.

Key terms: monitoring, observability, telemetry, structured telemetry, logs, metrics, traces, spans, context-aware AI systems, infrastructure traces

8.3.2 Active telemetry

This section explains how active telemetry turns context-rich runtime events into decisions and evaluation signals, then uses a support-assistant incident to show how traces reveal the causes of a wrong answer.

Key terms: active telemetry, semantic metadata, bidirectional feedback, dynamic sampling and enrichment, human-readable semantics, model-graded evaluations, model span, retrieval span, session logs, stale fallback index

8.3.3 Context graphs

This section explains how context graphs reveal relationships behind AI failures and support incident review without requiring a graph database.

Key terms: context graph, nodes, edges, relational failures, superseded_by, orchestration race, groundedness score, table of triples, blast radius, failure grouping

8.3.4 Observability tools

This section explains how standardized telemetry and AI-specific tracing tools work together, and what a robust observability stack must correlate and protect.

Key terms: OpenTelemetry, GenAI semantic conventions, traces, metrics, general backends, dedicated AI observability platforms, LLM-specific payload context, execution traces, prompt versions, retrieval indexes

8.4 Hands-on

Introduces hands-on evaluation and observability examples and points readers to the companion repository for complete scripts and setup instructions.

8.4.1 Metrics with DeepEval

This section presents DeepEval as a way to score one LLM interaction with multiple metrics while keeping evaluation logic close to the test case.

Key terms: DeepEval, AnswerRelevancyMetric, FaithfulnessMetric, GEval, regression suites

8.4.2 Evals with Promptfoo

This section presents Promptfoo as a config-driven tool for comparing prompts and checking LLM application behavior.

Key terms: Promptfoo, prompt variants, test set, model-graded rubric, deterministic assertion, prompt iteration, regression checks, config-driven, audit

8.4.3 LLM-as-judge with Ragas

The section uses a small RAG evaluation example to show what Ragas judge scores can signal and why they need calibration before guiding releases.

Key terms: LLM-as-judge, Ragas, faithfulness, answer relevancy, context precision, context recall, decision-support data, deterministic checks, human-labeled examples, release gates

8.4.4 Observability with Langfuse

This section shows how a Langfuse callback handler records a small agent’s execution for observability and debugging.

Key terms: Langfuse, callback handler, trace, spans

8.4.5 Observability with LangSmith

This section uses LangSmith to show how tracing context assembly supports observability, then connects runtime traces to an evaluation-and-improvement feedback loop.

Key terms: LangSmith, context assembly, trace metadata, active telemetry, evaluation, observability, eval cases, traces, feedback loop

8.5 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 7: Context management and orchestration Chapter 9: Governance and operations →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.