AI feedback is evaluative information generated by an AI system about the quality of an input, model output, or action. It can create or augment data for training and evaluation, most commonly by judging which of two candidate responses is preferable. This makes feedback generation cheaper and easier to scale than a pipeline relying entirely on human preference labels. section 12.4 section 12.4.1
Reinforcement learning from AI feedback (RLAIF) is a specific use of AI feedback in the RLHF pipeline: an AI produces preference data that can train a reward model and guide subsequent optimization. AI feedback is broader than RLAIF; the same judge infrastructure can support scalable evaluation and rubric-based rewards, including tasks without clearly verifiable answers. section 12.4 section 12.7
A practical feedback pipeline uses AI at several points:
- Generate preference labels for candidate completions, replacing or augmenting human comparisons used in reward-model training. section 12.4
- Evaluate model outputs at large scale with an LLM acting as a judge. section 12.4
- Score completions against per-prompt rubrics to supply rewards for RL-style training in domains such as scientific reasoning, creative writing, and model personality. section 12.7
- Combine AI labels with human labels by routing challenging examples to people while assigning most examples to AI feedback. section 12.4.1
The main trade-off is not simply quality versus cost. AI feedback can be generated for less than $0.01 per item with a frontier model, compared with roughly $1 or more for a human preference item, enabling experiments that human-labeling budgets may exclude. But the suitable balance depends on domain and evaluation breadth: evidence does not establish where AI labels consistently outperform human labels across chat, safety, reasoning, and mathematics. section 12.4.1
Synthetic preference data is characterized as lower-noise but higher-bias than human data. That can make AI feedback straightforward to start with, while creating a risk of systematic second-order effects. Human data remains important at capability frontiers and for ground-truth answers, so AI feedback is a scalable complement rather than a universally sufficient substitute. section 12.7
Hybrid preference labeling
For an RLHF run, collect candidate responses from a model checkpoint close to the one being optimized, then send most comparisons to an AI judge and route difficult cases to human annotators. Using data from the current family of models keeps preferences focused on the generation patterns that matter during optimization. section 12.4.1 section 11.2
Constitutional AI was an early large-scale implementation that helped establish RLAIF: written principles guided instruction-data revision and synthetic preference generation. More generally, strong GPT-4-class or better models made LLM-as-a-judge feedback and scoring sufficiently robust and consistent for modern synthetic-data pipelines. section 12.4 section 12.1
