liveBook · Manning

Ensemble

As explained in AI Model Evaluation

An evaluation setup that combines judgments from multiple evaluators, models, runs, or prompts for a single assessment.

An ensemble combines multiple judge calls into one verdict instead of trusting a single opinion. It reduces dependence on one model’s biases and exposes uncertainty that a single run can hide. section 10.1

Common ways to construct an ensemble include:

  • Repeated sampling runs the same prompt and model multiple times, surfacing variance hidden by a deterministic run. section 10.1
  • Multiple models send the same task to different model families, reducing dependence on one provider’s preferences. section 10.1
  • Multiple prompts keep the model fixed while varying the judging approach; disagreement may reveal ambiguity in the rubric. section 10.1

An ensemble should produce both an aggregated verdict and an agreement measure such as agreement rate or score variance. High agreement can proceed automatically; substantial disagreement should route the case to human review rather than being averaged away. section 10.1.1

Choosing the judging budget

Use a panel when gating a model launch, evaluating a contested safety decision, or calibrating a new judge. For continuous, low-stakes production checks, use a validated single judge and reserve panels for periodic audits or canary checks. section 10.1.2

How AI Model Evaluation covers it

How deep it goes
Details construction methods, aggregation, disagreement routing, and cost trade-offs, while deferring broader validation and escalation patterns.
Examples
Uses repeated samples, distinct models, varied prompts, launch gates, safety judgments, production checks, and periodic audits.
What it assumes
Assumes a working single judge and an evaluation task with a rubric or scoring procedure.
What you can do after
Operate a judge panel that aggregates verdicts, measures agreement, and routes uncertain evaluations for deeper review.
How it connects
Connects LLM-as-a-judge with majority vote, human review, abstention, escalation, and canary checks.

Also explained in

Compare how 16 books explain ensemble →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.