An ensemble is a collection of separately trained models whose predictions are combined or compared to improve information or reliability.
An ensemble combines separately trained reward models to estimate uncertainty through disagreement. Models initialized differently and trained on different label subsets expose pairs that need human judgment. section 7.4.2
Use ensemble disagreement to make preference-labeling more informative:
Train three reward models on slightly different preference-data subsets. section 7.4.2
Compare their predicted preferences or reward values for trajectory pairs. section 7.4.2
Prioritize pairs with split decisions or high prediction variance; skip pairs on which the models agree. section 7.4.2
How LLM Evaluation and Alignment, The Foundational Ideas covers it
How deep it goes
Explains ensemble construction and query selection, but stops short of implementation code or uncertainty formulas.
Examples
Uses simulated MuJoCo backflips and Atari games to illustrate labeling efficiency.
What it assumes
Assumes familiarity with reward models, policy trajectories, and human preference labels.
What you can do after
Apply ensemble disagreement to prioritize preference comparisons under a limited labeling budget.
How it connects
Connects reward models with human preferences, active learning, trajectory sampling, and reinforcement learning from human feedback.
Outlier Detection in PythonAn ensemble is a combination of multiple models or detectors whose outputs are combined to obtain more robust results than an individual model or detector.
Experimentation for EngineersA model ensemble is a collection of predictive models whose predictions or decisions are combined, compared, or otherwise used collectively.
Build AI Drug Discovery PipelinesEnsemble learning combines multiple individual predictive models to form a composite model with improved robustness or accuracy.
Deep Learning with Python, Third EditionA technique that combines multiple models or views to improve performance by exploiting complementary errors or representations.
Generative AI in ActionA model ensemble is an architecture that combines multiple machine-learning models to improve overall capability, performance, or reliability.
random forestincludes · A random forest is an ensemble learning model that aggregates predictions from multiple decision trees trained with randomized data samples, feature subsets, or both.
baggingincludes · Bagging, or bootstrap aggregation, is an ensemble technique that trains multiple models on bootstrap samples of data and combines their predictions.
gradient boostingincludes · Gradient boosting is an ensemble-learning technique that sequentially adds predictive models fitted to reduce a loss by following its gradient.
decision treeuses · A decision tree is a tree-structured model or representation in which successive tests or choices lead through branches to outcomes, predictions, or actions.
diversityuses · Diversity is the degree of variation among members of a set or population in characteristics relevant to that set or population.
language modeluses · A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.