An evaluation metric measures how well an AI system meets a specified criterion. Its score can potentially become a training signal, but a score is not automatically effective for training. section 1.3.1 section 7.0
Choose a metric with the task in mind. Verifiable tasks have objectively correct answers; open-ended language tasks may have several valid answers. A word-overlap score can miss equivalent meanings expressed with different words. section 1.1 section 2.0
Use metrics in an iterative evaluation and alignment cycle:
- Define what good performance means and choose measurable criteria. section 1.3.1
- Measure the system, then examine errors to identify recurring failures. section 1.3.1
- Change the system to address those failures and evaluate it again. section 1.3.1
Customer service chatbot
Measure order-status accuracy, politeness, response time, and resolution rate across test conversations. If inaccurate answers often invent shipping dates when database values are missing, adjust the prompt and retrieve shipping policies, then remeasure accuracy. section 1.3.1
A simple observable proxy can support fast iteration without capturing every aspect of quality. BLEU compares a translation with human references using lexical overlap; ROUGE uses similar overlap principles for summaries, emphasizing recall. Neither captures meaning or paraphrasing reliably. section 2.1.1 section 2.0
Some evaluation metrics produce scalar scores like the signals reinforcement learning algorithms consume when updating a policy. Whether a particular score works as a training signal still requires investigation. section 7.0









































