liveBook · Manning

Task definition

As explained in Evaluating AI Systems

A task definition specifies the activity an AI system is expected to perform and the conditions that define its scope and success.

A task definition states what an AI system is expected to do, where its responsibilities stop, and what counts as success. A medical chatbot described as helping patients “manage their health” may be judged against personalized advice it was never designed or permitted to give. section 2.4.1

Two mismatches can distort evaluation. A scope mismatch expects a system to do more than its intended task; an equivalence mismatch treats similar-looking tasks as interchangeable, such as health education and medical triage. Vague correctness criteria create another problem: evaluators may disagree about whether an answer needs a safety warning or should avoid mentioning rare conditions. section 2.4.1

Check whether an evaluation dataset reflects the task you intend to assess:

  • Compare task definitions, assumptions, edge cases, labels, and success criteria. section 4.5.1
  • Inspect a diverse sample for situations your system must handle, including messy or emotionally loaded queries when relevant. section 4.5.1
  • Test a baseline that reflects your intended behavior; poor dataset scores despite appropriate behavior can reveal a mismatch. section 4.5.1
Figure 2.3 Two common sources of task-definition failure. In a scope mismatch, the system is expected to perform tasks it was never designed to handle while in an equivalence mismatch, stakeholders assume that solving one task is equivalent to solving a different one.

Use the comparison to distinguish systems asked to exceed their scope from systems evaluated as though they perform a different task. section 2.4.1

How Evaluating AI Systems covers it

How deep it goes
Explains scope, equivalence, and success-criteria failures, then gives practical checks for dataset fit.
Examples
Uses medical-chatbot scope and triage contrasts, plus summarization and conversational-helpfulness dataset scenarios.
What it assumes
Requires an intended task definition for comparison with dataset goals, assumptions, edge cases, labels, and success criteria.
What you can do after
Evaluate dataset fit by comparing task definitions, inspecting samples, and testing a baseline.
How it connects
Connects task definitions to evaluation datasets, success criteria, and distinctions between health education and medical triage.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.