liveBook · Manning

Objective function

As explained in LLM Evaluation and Alignment, The Foundational Ideas

An evaluation metric is a measurable quantity that represents how well an AI system satisfies a specified criterion.

An evaluation metric measures how well an AI system meets a specified criterion. Its score can potentially become a training signal, but a score is not automatically effective for training. section 1.3.1 section 7.0

Choose a metric with the task in mind. Verifiable tasks have objectively correct answers; open-ended language tasks may have several valid answers. A word-overlap score can miss equivalent meanings expressed with different words. section 1.1 section 2.0

Use metrics in an iterative evaluation and alignment cycle:

  • Define what good performance means and choose measurable criteria. section 1.3.1
  • Measure the system, then examine errors to identify recurring failures. section 1.3.1
  • Change the system to address those failures and evaluate it again. section 1.3.1

Customer service chatbot

Measure order-status accuracy, politeness, response time, and resolution rate across test conversations. If inaccurate answers often invent shipping dates when database values are missing, adjust the prompt and retrieve shipping policies, then remeasure accuracy. section 1.3.1

A simple observable proxy can support fast iteration without capturing every aspect of quality. BLEU compares a translation with human references using lexical overlap; ROUGE uses similar overlap principles for summaries, emphasizing recall. Neither captures meaning or paraphrasing reliably. section 2.1.1 section 2.0

Some evaluation metrics produce scalar scores like the signals reinforcement learning algorithms consume when updating a policy. Whether a particular score works as a training signal still requires investigation. section 7.0

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains metric selection, error analysis, and re-evaluation, while noting that a scalar score may not work as a training signal.
Examples
Illustrates chatbot accuracy, politeness, latency, and resolution metrics alongside BLEU and ROUGE for translation and summarization.
What you can do after
Evaluate/understand AI behavior by choosing criteria, interpreting metric limitations, and investigating failures before making changes.
How it connects
Connects evaluation metrics to reinforcement learning, human feedback, lexical overlap, and semantic similarity.

Also explained in

Compare how 42 books explain objective function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.