An ensemble combines multiple judge calls into one verdict instead of trusting a single opinion. It reduces dependence on one model’s biases and exposes uncertainty that a single run can hide. section 10.1
Common ways to construct an ensemble include:
- Repeated sampling runs the same prompt and model multiple times, surfacing variance hidden by a deterministic run. section 10.1
- Multiple models send the same task to different model families, reducing dependence on one provider’s preferences. section 10.1
- Multiple prompts keep the model fixed while varying the judging approach; disagreement may reveal ambiguity in the rubric. section 10.1
An ensemble should produce both an aggregated verdict and an agreement measure such as agreement rate or score variance. High agreement can proceed automatically; substantial disagreement should route the case to human review rather than being averaged away. section 10.1.1
Choosing the judging budget
Use a panel when gating a model launch, evaluating a contested safety decision, or calibrating a new judge. For continuous, low-stakes production checks, use a validated single judge and reserve panels for periodic audits or canary checks. section 10.1.2















