Calibration data is a human-rated response set used to check whether an AI judge's scores or preferences reflect intended quality judgments. It should match the task or production prompt distribution and cover meaningful quality variation. section 4.4.4 section 8.3.2
Build and use calibration data through these steps:
- Sample responses from the actual evaluation or production distribution; larger preference-calibration exercises may use 500–1,000 prompts with two responses per prompt. section 8.3.2
- Collect ratings or pairwise rankings from two or three calibrated human labelers, then measure human–human agreement as the comparison ceiling. section 8.3.2
- Run the AI judge with the same prompt, constitution, and reasoning structure intended for production, then compare its agreement with human consensus. section 8.3.2
- For numeric judges, map raw scores to human ratings; for unreliable absolute scores, consider percentile ranking or a human-validated pass/fail threshold. section 4.4.4
Restoring a compressed score range
Suppose a judge assigns scores from 3.5 to 4.8 while humans rate the same responses from 1 to 5. A calibration set spanning the full quality range can define a mapping that spreads the judge's outputs across the human scale. Recalibrate when monitoring detects drift. section 4.4.4





