An evaluation criterion is a defined standard for judging an AI system’s behavior. It specifies what “good” looks like before measurement, error analysis, and alignment. section 1.3.1
A practical evaluation cycle uses criteria to:
- Define desired behavior for a specific task rather than relying on general benchmarks. section 1.3.1
- Measure current performance, inspect failures, and align the system against the observed gap. section 1.3.1
Customer-service chatbot
A good response can be judged for factual order-status accuracy, politeness, response time under two seconds, and issue resolution without escalation. A 500-conversation evaluation produced scores of 87% accuracy, 94% politeness, 1.8-second average latency, and 72% resolution; error analysis found hallucinated shipping dates when database values were null. section 1.3.1
Figure 1.10 The iterative cycle of AI engineering. Define establishes what success looks like, Evaluate measures the current system against the designed evaluation metrics, Error analysis identifies failure modes, and then Align optimizes the system using different methods. The cycle then repeats.
Use the iterative-cycle figure to navigate from defining criteria through evaluation, error analysis, alignment, and repeated re-evaluation. section 1.3.1



