An execution-based metric assesses how well an AI system performs when its outputs are executed, used, or acted upon in a workflow or environment. It is especially useful for agents whose behavior unfolds over multiple steps, where evaluation can track task success or goal completion. section 3.2
Resolving a billing dispute
A customer-support agent can be evaluated on whether it resolves a dispute by retrieving account information, identifying the issue, applying the appropriate policy, and updating the customer record. section 3.2
Execution-based metrics capture end-to-end behavior and interactions between components that output-level metrics may miss. They can align closely with user and business outcomes, but typically require specialized infrastructure. Choose an outcome that reflects what users need: for a medical chatbot, that might be whether users are guided to appropriate next steps. section 3.2
