liveBook · Manning

Ensemble

As explained in LLM Evaluation and Alignment, The Foundational Ideas

An ensemble is a collection of separately trained models whose predictions are combined or compared to improve information or reliability.

An ensemble combines separately trained reward models to estimate uncertainty through disagreement. Models initialized differently and trained on different label subsets expose pairs that need human judgment. section 7.4.2

Use ensemble disagreement to make preference-labeling more informative:

  • Train three reward models on slightly different preference-data subsets. section 7.4.2
  • Compare their predicted preferences or reward values for trajectory pairs. section 7.4.2
  • Prioritize pairs with split decisions or high prediction variance; skip pairs on which the models agree. section 7.4.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains ensemble construction and query selection, but stops short of implementation code or uncertainty formulas.
Examples
Uses simulated MuJoCo backflips and Atari games to illustrate labeling efficiency.
What it assumes
Assumes familiarity with reward models, policy trajectories, and human preference labels.
What you can do after
Apply ensemble disagreement to prioritize preference comparisons under a limited labeling budget.
How it connects
Connects reward models with human preferences, active learning, trajectory sampling, and reinforcement learning from human feedback.

Also explained in

Compare how 16 books explain ensemble →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.