Model-based evaluation judges model outputs with an evaluator model instead of directly using a human judge. A reward model scores responses according to a learned estimate of human preferences. section 10.19
This approach makes large-scale comparisons faster and cheaper than human evaluation, but it remains an imperfect proxy: reward models can encode subjective judgments and become unreliable when optimized as targets. section 10.19
A responsible evaluation setup should:
- Specify evaluation details, because formatting and other minor setup changes can shift results. section 10.19
- Measure both helpfulness and safety rather than treating response quality or accuracy as the only concern. section 10.19
- Use general or diverse reward models when possible, rather than evaluating with the same measure used as a training target. section 10.19
