An evaluation activity is a step in systematically assessing how an AI system behaves in its intended context. Together, these activities support decisions about whether and how to use a system—not merely whether its model scores well on a benchmark. section 1.1 section 1.8
Evaluation should consider the whole system, including its interface, data pipelines, human interaction, and policies. Results depend on who uses the system, how it fits into a workflow, and whether evaluation data reflects real conditions. section 1.8 section 1.4
Figure 1.1 The difference between an AI model and an AI system. A model is a self-contained computational component that transforms inputs into outputs, while an AI system combines the model with s user interfaces, data pipelines, integration logic, human-in-the-loop mechanisms, and other functionalities and features to perform real-world tasks.
The distinction between a model and a system helps establish what an evaluation needs to cover. section 1.2
An evaluation project typically iterates through six activities, in order:
- Frame and scope the evaluation: establish what is being evaluated and what decision the evaluation should inform. section 1.5 section 1.8
- Choose metrics: select measures that reflect the relevant tasks and dimensions, rather than relying on accuracy alone. section 1.5 section 1.8
- Assemble evaluation data: use data suited to the intended purpose and context. section 1.4 section 1.8
- Run the evaluation and compute results. section 1.8
- Interpret and communicate results to the people who need them. section 1.8
- Decide and act on the evidence, using judgment to weigh trade-offs and choose next steps. section 1.8
Evaluating a legal advice chatbot
Testing legal fact accuracy alone can miss whether the chatbot handles ambiguity, communicates uncertainty, helps users navigate procedures, or refers them to a lawyer when appropriate. A safety-focused evaluation may also need edge cases and adversarial prompts; a decision to launch a beta can call for a different design. section 1.5
Evaluation is not a one-time checklist. AI behavior can vary with inputs and deployment context, and systems may degrade or encounter unforeseen failures. Continuous evaluation helps avoid both premature trust and the dismissal of potentially useful systems. section 1.4 section 1.3
