liveBook · Manning

Combined evaluation

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Combined evaluation assesses multiple quality dimensions within a single judge prompt or evaluation call.

Combined evaluation scores several quality dimensions in one judge prompt or evaluation call. It improves efficiency when dimensions are relatively independent, but requires prompts that clearly separate and independently assess each dimension. section 4.4.3

Design a combined evaluation by following these checks:

  • Name each dimension separately, such as helpfulness, accuracy, tone, and policy compliance. section 4.4.3
  • Instruct the judge to assess dimensions independently; generating reasoning steps for each dimension can help maintain separation. section 4.4.3
  • Prefer sequential evaluation when dimensions interact complexly or need specialized prompts; prefer combined evaluation when minimizing cost matters. section 4.4.3
  • Weight dimensions by their impact instead of averaging them naively; use a geometric mean when principled weights are unavailable and severe failures should dominate. section 4.4.3

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains prompt separation, method selection, reasoning support, and weighted score combination, but does not provide a full implementation.
Examples
Evaluates a customer service response for helpfulness, accuracy, tone, and company-policy compliance.
What you can do after
Apply it via a structured judge prompt and combine dimension scores without letting severe failures disappear through averaging.
How it connects
Contrasts with sequential evaluation and connects to evaluation rubrics, G-Eval, geometric means, and arithmetic means.

Also explained in

Compare how 2 books explain combined evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.