An AI agent handles a request through a chain of decisions, such as choosing a model, calling a tool, searching a source, or asking a clarifying question. In a recommendation system, these steps make the agent’s behavior harder to evaluate than a single choice of which movie to show. section 5.9.1
Choose evaluation measures to match the user’s goal. For a conversational movie search, task completion, constraint adherence, and tool-call accuracy can show whether the agent found a suitable title, respected requests such as “nothing scary,” and used the right retrieval or ranking tool. A completed task alone can mislead: an agent may take actions that users need to undo or correct. section 1.3.5 section 8.3.1
Evaluate the chain carefully: asking what would have happened if the agent acted differently raises the question of which decision to change. Assessing the entire chain at once is usually a mistake. For internal support agents, replaying historical tickets can provide an offline setting for checking relevance, factual consistency, policy adherence, and agreement with human reviewers. section 5.9.1 section 1.3.2













































