liveBook · Manning

Pareto frontier

As explained in LLM Evaluation and Alignment, The Foundational Ideas

The set of solutions for which no objective can improve without worsening at least one conflicting objective.

A Pareto frontier is the set of optimal trade-offs between conflicting metrics: improving one objective requires worsening at least one other. It matters when no single score captures user satisfaction or system quality. section 1.1.2

Use a frontier when evaluation involves competing signals:

  • Combine automated metrics, sampled human evaluation, LLM-as-a-Judge, production A/B tests, and error analysis rather than relying on one score. section 1.1.2
  • Recognize that different users may prefer different points on the curve. section 1.1.2
  • Treat offline metrics as incomplete evidence and continue monitoring deployed systems. section 1.1.2

Metric optimization can miss user needs

A chatbot optimized for BLEU may sound robotic, while a summarizer maximizing ROUGE may omit important information absent from the reference dataset. These systems may need different positions on a quality frontier. section 1.1.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines optimal trade-offs and situates them among evaluation methods without developing a formal optimization procedure.
Examples
Uses BLEU-optimized chatbots and ROUGE-optimized summarizers to expose metric–user preference conflicts.
What you can do after
Evaluate systems across competing quality signals and interpret metric results against user preferences and production performance.
How it connects
Connects automated metrics, human evaluation, LLM-as-a-Judge, A/B testing, error analysis, and production monitoring.

Also explained in

Compare how 5 books explain Pareto frontier →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.