Active learning selects examples for labeling by expected information value. In preference learning, ensemble disagreement identifies trajectory pairs most likely to improve the reward model. section 7.4.2
A practical active-querying loop:
- Train several reward models on different label subsets. section 7.4.2
- Select trajectory pairs in proportion to ensemble disagreement. section 7.4.2
- Send selected pairs for human preference judgments and update the preference dataset and policy. section 7.4.2
Small-budget preference collection
Agreement among three reward models provides little information, while split decisions or high predicted-reward variance identify useful queries. Reported budgets were roughly 900 queries for simulated MuJoCo backflips and 5,500 for Atari games. section 7.4.2
Figure 7.5 The three concurrent processes in Christiano et al.'s 2017 system. The policy generates trajectory segments that feed a preference-labeling queue. Human annotators compare pairs and return binary judgments, which accumulate in the preference dataset. The reward predictor is trained on the dataset, and its scores drive the policy-gradient updates. Arrows show the data flow; all three loops run continuously rather than in sequence.
Trace the concurrent flow from policy-generated trajectories through human comparisons and reward prediction to policy updates. section 7.4.2





