8 Evaluation and observability
This chapter covers
- A taxonomy of context failures and degradation modes in LLM-based systems
- Evaluation workflows, automated graders, and evaluation-driven development
- Quality metrics for retrieval pipelines and multi-step agentic systems
- Synthetic datasets, industry benchmarks, and LLM-as-judge scoring
- Observability, active telemetry, and context graphs for runtime tracing
Chapter 1 introduced the context engineering stack as three functional layers (sources, management, and orchestration) supported by cross-cutting assurance disciplines. This chapter focuses on the first two: evaluation and observability. These disciplines provide the feedback loops, automated graders, and telemetry required to diagnose context degradation, assess retrieval quality, and keep multi-step agent behavior aligned with user intent. The discussion begins with a systematic taxonomy of context failures, explores evaluation methods, quality metrics, and concludes with runtime observability, active telemetry, and context graphs.
8.1 A taxonomy of context failures
Context failures occur when the information supplied to an LLM-based system does not support the task, user intent, or the agent’s expected behavior. The model may produce a correct answer, but the surrounding context can make that answer unlikely. The context may be outdated, noisy, excessive, contradictory, adversarial, or poorly structured.