A Pareto frontier is the set of optimal trade-offs between conflicting metrics: improving one objective requires worsening at least one other. It matters when no single score captures user satisfaction or system quality. section 1.1.2
Use a frontier when evaluation involves competing signals:
- Combine automated metrics, sampled human evaluation, LLM-as-a-Judge, production A/B tests, and error analysis rather than relying on one score. section 1.1.2
- Recognize that different users may prefer different points on the curve. section 1.1.2
- Treat offline metrics as incomplete evidence and continue monitoring deployed systems. section 1.1.2
Metric optimization can miss user needs
A chatbot optimized for BLEU may sound robotic, while a summarizer maximizing ROUGE may omit important information absent from the reference dataset. These systems may need different positions on a quality frontier. section 1.1.2




