liveBook · Manning

AI agent

As explained in AI Model Evaluation

An agentic system is an AI system that performs tasks through multi-step actions, often by deciding when and how to invoke tools.

An AI agent handles a request through a chain of decisions, such as choosing a model, calling a tool, searching a source, or asking a clarifying question. In a recommendation system, these steps make the agent’s behavior harder to evaluate than a single choice of which movie to show. section 5.9.1

Choose evaluation measures to match the user’s goal. For a conversational movie search, task completion, constraint adherence, and tool-call accuracy can show whether the agent found a suitable title, respected requests such as “nothing scary,” and used the right retrieval or ranking tool. A completed task alone can mislead: an agent may take actions that users need to undo or correct. section 1.3.5 section 8.3.1

Evaluate the chain carefully: asking what would have happened if the agent acted differently raises the question of which decision to change. Assessing the entire chain at once is usually a mistake. For internal support agents, replaying historical tickets can provide an offline setting for checking relevance, factual consistency, policy adherence, and agreement with human reviewers. section 5.9.1 section 1.3.2

How AI Model Evaluation covers it

How deep it goes
Traces chained decisions and evaluation challenges, but does not specify an agent implementation or full evaluation procedure.
Examples
Uses conversational movie search and internal support agents drafting replies or summarizing ticket threads as evaluation settings.
What you can do after
Evaluate agents with task completion, constraint adherence, and tool-call accuracy while checking whether measured success matches user outcomes.
How it connects
Connects AI agents with recommendation systems, counterfactual evaluation, offline evaluation, and reward hacking.

Also explained in

Compare how 46 books explain AI agent →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.