An evaluation criterion is a stated condition or standard used to judge whether a system meets a requirement. It connects what is assessed to a decision: for example, whether a medical chatbot’s unsafe recommendations remain below an acceptable threshold before deployment. section 2.2.3 section 5.2.1
A criterion is related to, but distinct from, an evaluation dimension and a metric. A dimension identifies an aspect to assess, such as safety; a metric specifies how to measure it, such as the rate of unsafe responses. A decision criterion can then state what level is acceptable for the intended use and its risks. A reported error rate alone does not establish whether a system is ready for use. section 2.5 section 2.2.3
Choose criteria after agreeing on the evaluation’s purpose, users, tasks, and assumptions. Focus on dimensions that inform a concrete action rather than trying to assess everything at once. A launch decision, a comparison with a previous version, and an investigation of safety risks may call for different criteria and evaluation approaches. section 2.0 section 2.5 section 1.5
Medical chatbot readiness
A go/no-go evaluation could check whether unsafe recommendations stay below an acceptable threshold, required disclaimers and escalation behaviors are reliably triggered, and performance remains stable under realistic load. The acceptable limits depend on intended use, operational constraints, obligations, and stakeholder risk tolerance. section 2.2.3
Match the form of judgment to the criterion. A binary check works when the condition is clear-cut, such as whether a chatbot mentions a specific drug contraindication. Clarity for a lay patient or an appropriately cautious tone can admit degrees; a scalar judgment captures that nuance but can be interpreted inconsistently. For a complex quality judgment, several well-defined binary checks may offer a more interpretable alternative. section 5.2.1
Figure 5.2 Replacing a subjective scalar judgment with a set of structured binary checks. This approach often yields more reliable and interpretable judgments for complex evaluation criteria.

Illustrates how a broad subjective rating can be replaced with structured binary checks. section 5.2.1
Criteria should reflect the system’s actual role and context, not just an easy-to-measure output. A legal chatbot may need to navigate procedures, handle ambiguity, and refer users to a lawyer; factual accuracy alone would miss important aspects of usefulness and safety. section 1.5



