A dry-run tool adapter lets shadow evaluations exercise an agent’s tool-calling pipeline without causing externally visible side effects. It prevents duplicate emails, record changes, tickets, workflows, and third-party costs while retaining realistic processing. section 4.3.3
Choose an interception strategy based on the tool and the realism required:
- Route calls to sandboxed APIs or isolated data stores when the evaluation needs realistic interactions. section 4.3.3
- Use read-only credentials, recorded responses, or mock outputs when external execution is unnecessary. section 4.3.3
- Capture outbound actions with sink components, especially for emails, notifications, and other user-visible effects. section 4.3.3
Figure 4.2 illustrates how shadow traffic adds additional load to the supporting infrastructure such as feature database layer, metadata services, caches and user context lookups. This is one of the primary risks of shadow traffic that's worth over communicating so that the machine learning or platform infrastructure teams aren’t surprised by an increase in requests as a result of the shadow traffic evaluation.
Use the infrastructure-load figure to account for additional requests that shadow traffic sends to feature stores, metadata services, caches, and user-context lookups. section 4.3.3
