An evaluation goal is the purpose an evaluation serves and the decision its results should support. It turns a broad question such as “Is this system any good?” into a shared question about success, users, and assumptions. The goal shapes what to evaluate, which scenarios matter, and what evidence to collect. section 2.0 section 2.7
Different goals call for different evidence:
- Set expectations by identifying what the system can reliably do, where it struggles, and when its performance changes. This is useful for aligning stakeholders before external demonstrations or decisions. section 2.2.1
- Compare the system with a reference point, such as a previous version, a human baseline, or a performance target. Beating a comparator does not by itself establish that the system is safe or suitable for use. section 2.2.2
- Identify and mitigate risks by testing edge cases and adversarial prompts. This requires different evidence from an evaluation focused only on response accuracy or fluency. section 1.5
Evaluating a legal chatbot
A legal chatbot may need to provide factual answers, help users navigate procedures, and recognize when to refer someone to a lawyer. An accuracy-only evaluation could miss those needs. If the goal is risk mitigation, testing vague queries and adversarial prompts is more informative than relying only on canned legal questions. section 1.5
Goals can overlap, but trying to satisfy all of them at once can make an evaluation confusing and bloated. Agree on what matters now, given the decisions, resources, and risks, then select the tasks and performance dimensions that serve those priorities. Communicate results without implying that evidence of one quality, such as fluency, establishes another, such as safety. section 2.2 section 2.7 section 1.5
