An agent trajectory is the ordered record of a run: model calls, proposals, policy decisions, tool responses, and observed state changes. It connects what the agent attempted with what the system actually did, making the run available for diagnosis and review. section 9.5.1
A useful trace correlates the user, agent identity, workload, session, run, tool call, approval, and result across services. Preserve input references, versions, and identities alongside the sequence. Replaying a prompt alone is insufficient: model settings, session context, and external tool data may have changed. section 9.5.1
Table 9.7 The agent observability stack. Traditional observability extended with evaluations and replays.
|
Pillar
|
Traditional software
|
Agent systems
|
|
Logs
|
Application events, errors
|
Tool calls, model outputs, policy decisions
|
|
Metrics
|
Latency, throughput, error rate
|
Task outcomes, cost, overrides, tool selection accuracy
|
|
Traces
|
Request → service → database
|
Agent trajectory: context → proposal → policy → tool result → observed effect
|
|
Evaluations (new)
|
Unit tests, integration tests
|
Outcome tests, LLM-as-judge, human review
|
|
Replays (new)
|
Request or event replay
|
Replay stored trajectories with versioned context
|
Compare trajectories with logs, metrics, evaluations, and replay evidence; each answers a different question about a run. section 9.5.1
Diagnosing an excessive coupon
A successful coupon tool call does not establish that the compensation was authorized. Inspect the proposal, the messages the model saw, the case limit, earlier coupons, and the policy decision. An excessive proposal points toward input or planning; a gate that admitted it also implicates policy or task-state input. Turn the incident into regression cases for both excessive and valid requests, linking new traces and their versions to the incident. section 9.5.6
In collaborative runs, inspect the seams as well as the final result: rejected handoffs, duplicate work, contradictions, blocked dependencies, and integration rework can localize coordination failures. For consequential work, attach the claim, source, artifact version, acceptance criterion, decision owner, and integration receipt. A trace can reveal misalignment, but it cannot prove reviewer independence or final correctness. section 8.6
Build the execution record from observable behavior, not an assumed account of the model's reasoning. A trace can show what happened, but not prove why the model thought it was right. At production volume, use structured alerts and stratified sampling by task difficulty, tenant, and risk; high-risk paths may require complete retention. section 9.5.7 section 5.9
A raw trace can also support learning without becoming an instruction. Keep the execution, outcome, tool evidence, and environment version for audit; compress it into a candidate lesson with provenance, then admit a scoped heuristic only with sufficient support. A replay record alone grants no authority to execute a tool call or bypass a gate. section 7.5.1
























