chapter eight

8 Evaluation and observability

 

This chapter covers

  • A taxonomy of context failures and degradation modes in LLM-based systems
  • Evaluation workflows, automated graders, and evaluation-driven development
  • Quality metrics for retrieval pipelines and multi-step agentic systems
  • Synthetic datasets, industry benchmarks, and LLM-as-judge scoring
  • Observability, active telemetry, and context graphs for runtime tracing

Chapter 1 introduced the context engineering stack as three functional layers (sources, management, and orchestration) supported by cross-cutting assurance disciplines. This chapter focuses on the first two: evaluation and observability. These disciplines provide the feedback loops, automated graders, and telemetry required to diagnose context degradation, assess retrieval quality, and keep multi-step agent behavior aligned with user intent. The discussion begins with a systematic taxonomy of context failures, explores evaluation methods, quality metrics, and concludes with runtime observability, active telemetry, and context graphs.

8.1 A taxonomy of context failures

Context failures occur when the information supplied to an LLM-based system does not support the task, user intent, or the agent’s expected behavior. The model may produce a correct answer, but the surrounding context can make that answer unlikely. The context may be outdated, noisy, excessive, contradictory, adversarial, or poorly structured.

8.1.1 Context rot

8.1.2 Context staleness

8.1.3 Context drift

8.1.4 Context poisoning

8.1.5 Context pollution

8.1.6 Context distraction

8.1.7 Context confusion

8.1.8 Context clash

8.1.9 Context overflow

8.2 Evaluation

8.2.1 Evals

8.2.2 Evaluation metrics

8.2.3 Evaluation datasets

8.2.4 Benchmarks

8.2.5 LLM-as-judge

8.3 Observability

8.3.1 Traditional observability foundations

8.3.2 Active telemetry

8.3.3 Context graphs

8.3.4 Observability tools

8.4 Hands-on