liveBook · Manning

Evaluation object

As explained in Evaluating AI Systems

The AI system, subsystem, or component whose performance or properties are being assessed.

An evaluation object is the AI system or component whose performance or properties are being assessed. Choosing the object is part of framing an evaluation: a result about one component should not automatically be taken as a conclusion about the whole system. section 1.5 section 2.0

A clear scope connects the object to an evaluation goal, the tasks it must perform, and the quality dimensions that matter. These choices guide the metrics, data, methods, and tools used later. Rushing to benchmarks before agreeing on them can leave teams with results that do not answer the question they intended to ask. section 1.5 section 2.0

Evaluating a legal chatbot

For a risk analysis, the object might be the fully deployed chatbot rather than a single component. Evaluation could focus on eviction-related questions and assess correctness, robustness, privacy, and security. Fluent responses alone would not establish that the chatbot is safe to use. section 1.5

To define the object and its scope, ask:

  • What decision should the evaluation support, and does it require assessing the full system or a component? section 1.5
  • Which tasks and scenarios matter, to whom, and under what assumptions? section 2.0
  • Which quality dimensions reflect success or risk in that context? section 1.5 section 2.0

How Evaluating AI Systems covers it

How deep it goes
Distinguishes full-system from component evaluation and relates the choice to goals, tasks, and quality dimensions.
Examples
Uses a deployed legal chatbot answering eviction-related questions to illustrate risk-oriented system evaluation.
What you can do after
Evaluate how the chosen system or component aligns with the evaluation goal and intended decisions.
How it connects
Connects evaluation scope with evaluation metrics, evaluation data, tasks, and quality dimensions.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.