8 Evaluation and observability
This chapter covers
- How context failures appear in LLM-based systems and how to classify them.
- How evals, metrics, datasets, benchmarks, and LLM-as-judge support context-aware evaluation.
- How observability, active telemetry, and context graphs expose the causes of AI system behavior.
Chapter 1 introduced context engineering as a layered architecture for designing and building context-aware AI systems. This stack consists of three core technical layers (context sources, management, and orchestration) surrounded by a harness that includes evaluation, observability, governance, and operations. This chapter analyzes the evaluation and observability components of the harness. These components are not just external constraints but integral parts of the overall architecture. They provide the feedback, evidence, and control mechanisms that keep the core context components aligned with the application objective.
The chapter starts by naming common context failures, as teams cannot evaluate or observe problems they cannot describe. It then explains evaluation methods for context-aware systems, including evals, metrics, benchmarks, and LLM-as-judge. The second part turns to observability, active telemetry, and context graphs.