liveBook · Manning

Dry-run tool adapter

As explained in AI Model Evaluation

A dry-run tool adapter is an integration layer that simulates a tool invocation without performing its externally visible side effects.

A dry-run tool adapter lets shadow evaluations exercise an agent’s tool-calling pipeline without causing externally visible side effects. It prevents duplicate emails, record changes, tickets, workflows, and third-party costs while retaining realistic processing. section 4.3.3

Choose an interception strategy based on the tool and the realism required:

  • Route calls to sandboxed APIs or isolated data stores when the evaluation needs realistic interactions. section 4.3.3
  • Use read-only credentials, recorded responses, or mock outputs when external execution is unnecessary. section 4.3.3
  • Capture outbound actions with sink components, especially for emails, notifications, and other user-visible effects. section 4.3.3
Figure 4.2 illustrates how shadow traffic adds additional load to the supporting infrastructure such as feature database layer, metadata services, caches and user context lookups. This is one of the primary risks of shadow traffic that's worth over communicating so that the machine learning or platform infrastructure teams aren’t surprised by an increase in requests as a result of the shadow traffic evaluation.

Use the infrastructure-load figure to account for additional requests that shadow traffic sends to feature stores, metadata services, caches, and user-context lookups. section 4.3.3

How AI Model Evaluation covers it

How deep it goes
Defines sandboxing, read-only access, recorded responses, mock outputs, and sinks, but not adapter internals.
Examples
Uses isolated data stores, intercepted emails, recorded responses, mock outputs, and captured outbound actions.
What it assumes
Assumes shadow traffic and external tool calls; teaches side-effect prevention through adapter and environment choices.
What you can do after
Configure shadow evaluations that exercise tool calls without sending communications, modifying records, or triggering external workflows.
How it connects
Connects shadow traffic with sandboxed APIs, read-only credentials, mock outputs, sink components, and security standards.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.