liveBook · Manning

Evaluation criterion

As explained in LLM Evaluation and Alignment, The Foundational Ideas

An evaluation criterion is a defined standard against which an AI system’s behavior is judged.

An evaluation criterion is a defined standard for judging an AI system’s behavior. It specifies what “good” looks like before measurement, error analysis, and alignment. section 1.3.1

A practical evaluation cycle uses criteria to:

  • Define desired behavior for a specific task rather than relying on general benchmarks. section 1.3.1
  • Measure current performance, inspect failures, and align the system against the observed gap. section 1.3.1

Customer-service chatbot

A good response can be judged for factual order-status accuracy, politeness, response time under two seconds, and issue resolution without escalation. A 500-conversation evaluation produced scores of 87% accuracy, 94% politeness, 1.8-second average latency, and 72% resolution; error analysis found hallucinated shipping dates when database values were null. section 1.3.1

Figure 1.10 The iterative cycle of AI engineering. Define establishes what success looks like, Evaluate measures the current system against the designed evaluation metrics, Error analysis identifies failure modes, and then Align optimizes the system using different methods. The cycle then repeats.

Use the iterative-cycle figure to navigate from defining criteria through evaluation, error analysis, alignment, and repeated re-evaluation. section 1.3.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines criteria, demonstrates measured scores, and connects failures to corrective interventions.
Examples
Uses a customer-service chatbot evaluated for accuracy, politeness, latency, and issue resolution.
What you can do after
Evaluate/understand it by defining task-specific standards, measuring performance, diagnosing failures, and guiding alignment changes.
How it connects
Connects evaluation criteria to measurement, error analysis, alignment, model selection, prompt engineering, and retrieval-augmented generation.

Also explained in

Compare how 4 books explain evaluation criterion →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.