A proxy metric is an imperfect measurable quantity used to approximate a desired objective. In RLHF, a reward model’s score or a pairwise preference signal stands in for human satisfaction because human values cannot be written as a precise reward function. Optimization can improve the proxy while moving the model away from the underlying goal. section 11.0 section 14.1.1
Treat proxy metrics as useful training signals rather than ground truth:
- Measure the desired outcome separately; a proxy can improve while downstream evaluations or human judgments worsen. section 14.4
- Watch for gaming and bias, including verbosity, hedging, repetition, sycophancy, over-refusal, and rare-token artifacts. section 14.1.1 section 14.4
- Manage optimization pressure with better data, reward-model ensembles, KL penalties, or larger policy models. section 14.4
Verbose but unhelpful answers
A model may learn to produce confident, lengthy responses because those outputs receive high reward-model scores. The proxy metric rises, but actual helpfulness does not; the model has optimized the measurement rather than the intended quality. section 14.1.1
