liveBook · Manning

Damage limitation

As explained in Designing AI Agents

A control that limits the scope or potential impact of an agent's harmful or erroneous actions.

Damage limitation bounds the harm an agent can cause even when an action passes authorization and sandbox checks. An allowed action may still be wrong, or become harmful through repetition. The design question is the largest loss the system permits before someone can intervene. section 9.3.4 section 9.3

Use controls matched to the action’s consequences:

  • Rate limits cap actions per minute or hour, aggregating counts across tasks, tenants, resources, and sub-agents. section 9.3.4
  • Budget caps bound token, API, or infrastructure spending while retaining user and task metadata for attribution. section 9.3.4
  • Time locks queue irreversible actions with a cancellation window and revalidation before execution. section 9.3.4
  • Saga-style rollback runs compensating actions in reverse order after a later workflow step fails. Where compensation is not clean, as with email or some fund transfers, tighter earlier ceilings are necessary. section 9.3.4

Enforce a cumulative spending cap

A compensation task has a 100-yuan cap and has already issued 60 yuan. A further 50-yuan proposal must be denied because the projected total is 110 yuan, even if the individual payment would normally be approved. Use host-owned state for confirmed spending and outstanding reservations; check and reserve atomically before execution, then settle the reservation from the external result. This prevents concurrent proposals from spending the same remaining budget. section 9.6

Set different limits for different recovery paths. An operations agent might scale capacity within a quota, while taking an availability zone offline requires explicit confirmation. Greater trust can give an agent more room within its limits, but must not move the hard ceiling or permit crossing tenant boundaries or disabling an independent kill switch. section 9.3.4

Limits do not prove that an authorized action is correct: one permitted database write can still contain the wrong value. Use dry runs, validation, or staged releases to build evidence before commitment. Keep the gate, limiter, sandbox, and kill switch sufficiently independent; a shared process or credential can let one compromise defeat several layers. Test containment with fault injection and measure reachable resources, executions outside isolation, and recovery time. section 9.3.6 section 9.3

How Designing AI Agents covers it

How deep it goes
Details quantitative limits, intervention windows, compensation, cumulative budget enforcement, and containment failure modes.
Examples
Uses operational scaling and availability decisions alongside a 100-yuan compensation cap to illustrate different limits.
What you can do after
Operate and configure action limits, spending caps, cancellation windows, and compensating workflows to contain agent failures.
How it connects
Connects approval gates, sandboxing, saga rollback, and progressive commitment to a hard blast-radius ceiling.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Agents. The section links point to the parts of the book it draws on.