An artificial intelligence (AI) system is software designed to perform tasks that typically require human intelligence, such as perception, reasoning, decision-making, or language use. Its purpose may be to classify images, generate text, recommend products, assess risk, or support expert decisions. section 1.2
An AI model transforms inputs into outputs; an AI system combines the model with other components to perform a real-world task. Interfaces, data pipelines, integration logic, human interaction, and policies can all affect the system’s behavior and impact. section 1.2 section 1.8
Figure 1.1 The difference between an AI model and an AI system. A model is a self-contained computational component that transforms inputs into outputs, while an AI system combines the model with s user interfaces, data pipelines, integration logic, human-in-the-loop mechanisms, and other functionalities and features to perform real-world tasks.
The figure distinguishes a model from the surrounding components of an AI system. section 1.2
AI systems can be grouped broadly by what they do:
- Predictive systems make forecasts or classifications. section 1.2
- Generative systems create content. section 1.2
- Agentic systems take actions to achieve goals in an environment. section 1.2
These categories describe purpose rather than implementation. Systems can use symbolic rules, machine learning, or both; hybrid systems may combine models with business rules, ontologies, knowledge graphs, or reasoning engines. section 1.2
A chatbot’s advice and an airline’s policy
An Air Canada chatbot told a passenger that he could buy a regular ticket and later request a bereavement-fare refund, contrary to the airline’s policy. A tribunal held the airline accountable for information provided through its website and ordered compensation. The incident illustrates why evaluation must consider the system’s advice in its operational and policy context. section 1.0
Evaluation asks what a system does, how well it does it, and whether its behavior is acceptable in context—not just how its model works. This is difficult because AI behavior can be probabilistic and data-dependent, with edge-case failures and unforeseen outcomes. section 1.2 section 1.8
A structured evaluation iterates through six activities:
- Frame and scope the evaluation. section 1.8
- Choose metrics and assemble evaluation data. section 1.8
- Run the evaluation and compute results. section 1.8
- Interpret and communicate results, then decide and act. section 1.8
Automation can support evaluation, but people must still judge what to measure, how to weigh trade-offs, and what action to take. section 1.8

