liveBook · Manning

Execution-based metric

As explained in Evaluating AI Systems

A metric that assesses system performance through outputs being executed, used, or acted upon in a workflow, application, or environment.

An execution-based metric assesses how well an AI system performs when its outputs are executed, used, or acted upon in a workflow or environment. It is especially useful for agents whose behavior unfolds over multiple steps, where evaluation can track task success or goal completion. section 3.2

Resolving a billing dispute

A customer-support agent can be evaluated on whether it resolves a dispute by retrieving account information, identifying the issue, applying the appropriate policy, and updating the customer record. section 3.2

Execution-based metrics capture end-to-end behavior and interactions between components that output-level metrics may miss. They can align closely with user and business outcomes, but typically require specialized infrastructure. Choose an outcome that reflects what users need: for a medical chatbot, that might be whether users are guided to appropriate next steps. section 3.2

How Evaluating AI Systems covers it

How deep it goes
Defines the category and illustrates its advantages and infrastructure limitation without specifying an implementation procedure.
Examples
Illustrates billing-dispute resolution by a support agent and appropriate next-step guidance from a medical chatbot.
What you can do after
Evaluate whether an AI system achieves an intended outcome through its actions rather than judging its output alone.
How it connects
Connects agentic AI systems with task success rate and goal completion rate.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.