An observability harness connects an agent’s decisions, execution, and outcomes into a causal record for monitoring, diagnosis, evaluation, and audit. It preserves evidence across model calls, policy checks, tools, approvals, and external systems, even when those signals live in different backends. section 9.5 section 9.5.1
Replaying a prompt alone cannot reconstruct an agent run: model versions, sampling settings, session context, and tool responses may differ. Record the ordered trajectory together with the identities, input references, and versions needed to examine it later. Shared correlation identifiers connect the user, agent, workload, session, run, tool call, approval, and result. section 9.5.1
Table 9.7 The agent observability stack. Traditional observability extended with evaluations and replays.
|
Pillar
|
Traditional software
|
Agent systems
|
|
Logs
|
Application events, errors
|
Tool calls, model outputs, policy decisions
|
|
Metrics
|
Latency, throughput, error rate
|
Task outcomes, cost, overrides, tool selection accuracy
|
|
Traces
|
Request → service → database
|
Agent trajectory: context → proposal → policy → tool result → observed effect
|
|
Evaluations (new)
|
Unit tests, integration tests
|
Outcome tests, LLM-as-judge, human review
|
|
Replays (new)
|
Request or event replay
|
Replay stored trajectories with versioned context
|
The observability stack adds evaluations and replay artifacts to logs, metrics, and traces. section 9.5.1
Design the harness around the evidence that must survive a governed action:
- Set the trace boundary and correlation identifiers across participating services. section 9.5 section 9.5.1
- Choose recorded events and metrics, retention periods, access policies, and redaction rules. section 9.5
- Check whether inputs, permissions, actions, and outcomes can be reconstructed without rerunning the agent. Measure trace coverage, dropped spans, telemetry overhead, and time to root cause. section 9.5
Tracing a compensation decision
A compensation task has a 100-yuan cap and has already issued 60 yuan. A proposed 50-yuan payment would exceed the cap, so the policy denies it. The decision record retains the state snapshot, proposal, projected total, and rule; the observer links that decision to the observed effect. A sampled trace or model-written summary is not the authoritative ledger for the task state. section 9.6
Collection is only half the loop. Attribute a bad outcome to the responsible layer, change the tool, knowledge, specification, or planning mechanism as appropriate, and use regression evaluation and staged deployment to generate new evidence for governance decisions. The harness supports approval gates, blast-radius controls, and progressive commitment; it does not replace their enforcement. section 9.5.1 section 9.6
