liveBook · Manning

Observability harness

As explained in Designing AI Agents

A system that records and exposes execution traces for monitoring, diagnosis, and audit.

An observability harness connects an agent’s decisions, execution, and outcomes into a causal record for monitoring, diagnosis, evaluation, and audit. It preserves evidence across model calls, policy checks, tools, approvals, and external systems, even when those signals live in different backends. section 9.5 section 9.5.1

Replaying a prompt alone cannot reconstruct an agent run: model versions, sampling settings, session context, and tool responses may differ. Record the ordered trajectory together with the identities, input references, and versions needed to examine it later. Shared correlation identifiers connect the user, agent, workload, session, run, tool call, approval, and result. section 9.5.1

Table 9.7 The agent observability stack. Traditional observability extended with evaluations and replays.
Pillar
Traditional software
Agent systems
Logs
Application events, errors
Tool calls, model outputs, policy decisions
Metrics
Latency, throughput, error rate
Task outcomes, cost, overrides, tool selection accuracy
Traces
Request → service → database
Agent trajectory: context → proposal → policy → tool result → observed effect
Evaluations (new)
Unit tests, integration tests
Outcome tests, LLM-as-judge, human review
Replays (new)
Request or event replay
Replay stored trajectories with versioned context

The observability stack adds evaluations and replay artifacts to logs, metrics, and traces. section 9.5.1

Design the harness around the evidence that must survive a governed action:

  • Set the trace boundary and correlation identifiers across participating services. section 9.5 section 9.5.1
  • Choose recorded events and metrics, retention periods, access policies, and redaction rules. section 9.5
  • Check whether inputs, permissions, actions, and outcomes can be reconstructed without rerunning the agent. Measure trace coverage, dropped spans, telemetry overhead, and time to root cause. section 9.5

Tracing a compensation decision

A compensation task has a 100-yuan cap and has already issued 60 yuan. A proposed 50-yuan payment would exceed the cap, so the policy denies it. The decision record retains the state snapshot, proposal, projected total, and rule; the observer links that decision to the observed effect. A sampled trace or model-written summary is not the authoritative ledger for the task state. section 9.6

Collection is only half the loop. Attribute a bad outcome to the responsible layer, change the tool, knowledge, specification, or planning mechanism as appropriate, and use regression evaluation and staged deployment to generate new evidence for governance decisions. The harness supports approval gates, blast-radius controls, and progressive commitment; it does not replace their enforcement. section 9.5.1 section 9.6

How Designing AI Agents covers it

How deep it goes
Specifies trace boundaries, correlation, evidence types, retention, redaction, and operational measures, then connects incident attribution to regression evaluation.
Examples
Traces a voucher promise and a compensation proposal that would exceed a 100-yuan task cap.
What you can do after
Diagnose agent runs by reconstructing trajectories and attributing failures to the relevant system layer.
How it connects
Connects distributed tracing, policy decisions, approval gates, blast radius control, and progressive commitment.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Agents. The section links point to the parts of the book it draws on.