liveBook · Manning

Active learning

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Active querying is a data-collection technique that selects examples for labeling according to their expected information value.

Active learning selects examples for labeling by expected information value. In preference learning, ensemble disagreement identifies trajectory pairs most likely to improve the reward model. section 7.4.2

A practical active-querying loop:

  • Train several reward models on different label subsets. section 7.4.2
  • Select trajectory pairs in proportion to ensemble disagreement. section 7.4.2
  • Send selected pairs for human preference judgments and update the preference dataset and policy. section 7.4.2

Small-budget preference collection

Agreement among three reward models provides little information, while split decisions or high predicted-reward variance identify useful queries. Reported budgets were roughly 900 queries for simulated MuJoCo backflips and 5,500 for Atari games. section 7.4.2

Figure 7.5 The three concurrent processes in Christiano et al.'s 2017 system. The policy generates trajectory segments that feed a preference-labeling queue. Human annotators compare pairs and return binary judgments, which accumulate in the preference dataset. The reward predictor is trained on the dataset, and its scores drive the policy-gradient updates. Arrows show the data flow; all three loops run continuously rather than in sequence.

Trace the concurrent flow from policy-generated trajectories through human comparisons and reward prediction to policy updates. section 7.4.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains ensemble-based selection and its rationale, while noting that modern pipelines often favor uniform sampling and throughput.
Examples
Uses simulated MuJoCo backflips, Atari games, trajectory pairs, and three disagreeing reward models as examples.
What it assumes
Assumes familiarity with preference judgments, reward models, policies, and trajectory pairs.
What you can do after
Apply it to prioritize informative preference queries when labeling budgets are small.
How it connects
Connects human preferences, reward models, policy learning, reinforcement learning, and reinforcement learning from human feedback.

Also explained in

Compare how 6 books explain active learning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.