An evaluator, or AI labeler, is a language model that judges outputs against instructions and produces scores, explanations, or preference labels. It can provide a scalable signal for open-ended tasks where many responses are valid and word-overlap metrics miss qualities such as helpfulness. Its judgments are not ground truth: they need human calibration and auditing. section 4.0 section 4.3.5
An evaluator can compare an output with a reference answer or assess its quality without one. In alignment, it can compare two candidate responses and label the preferred one. Reinforcement learning from AI feedback (RLAIF) uses these AI-generated preferences to train a reward model, replacing per-comparison human labeling while retaining human rubric design and oversight. section 1.1.2 section 7.5.8
To create and use preference labels:
- Define the quality criterion in a written rubric. Include clear scoring instructions and examples so the evaluator knows what to judge. section 4.2.1
- For each prompt, generate two candidate responses. Give the evaluator the prompt, both responses, and the relevant rubric or principle; ask which response better satisfies it. section 8.2.4
- Calibrate on human-labeled comparisons from the intended distribution. If agreement is inadequate, revise the rubric or evaluator before scaling; continue auditing for systematic bias. section 7.5.8
- Use the resulting preferences to train a reward model, which supplies a signal for the alignment pipeline. section 7.5.8
A principle-based comparison
A labeler can receive a conversation, two proposed responses, and a principle such as “do not invent factual claims unsupported by the source.” It selects the response that better follows the principle. In Constitutional AI, such preferences supply reward-model data; critiques and revisions can also supply supervised training data. section 8.1.3 section 8.2.4
Check what the evaluator actually rewards. Position, verbosity, and preference for outputs from the same model family can distort judgments, and biases can interact. An evaluator may also fail when a task exceeds its own capabilities. Use its labels as one signal, not the sole basis for safety-critical decisions or final release; retain human review and other safeguards. section 4.3.5 section 4.4.5




















