Self-enhancement bias is a language-model judge’s tendency to favor answers produced by itself or by models from the same family, even when those answers are not objectively better. section 4.3.3
Diagnose the bias through controlled comparisons:
- GPT-4 judges favored GPT-4 responses by about 10% more than human evaluators, while Claude-v1 favored Claude responses by about 25%; GPT-3.5 showed no significant self-preference. section 4.3.3
- Self-preference remains detectable after controlling for response quality, but its magnitude varies by task type and model family. section 4.3.3
- Similar training data, reinforcement-learning procedures, diction, sentence structures, and reasoning styles may make familiar outputs appear better. section 4.3.3
Evaluation and RLHF risk
When GPT-4 compares a GPT-4 response with a Claude response, it may rate its own model family more favorably. In an RLHF pipeline, repeated preference for the judge’s style can optimize stylistic mimicry instead of genuine quality improvement. section 4.3.3 section 4.3.5
Figure 4.8 Self-enhancement bias varies across model families. Some judges (GPT-4, Claude) show elevated win rates for their own outputs, while others (GPT-3.5) do not exhibit significant self-preference.
Compare self-preference across GPT-4, Claude, and GPT-3.5 model families. section 4.3.3
Figure 4.9 Self-enhancement bias can create feedback loops in RLHF pipelines. If the judge prefers its own style, training may optimize for stylistic mimicry rather than genuine quality improvement.
Trace how judge preferences can create RLHF feedback loops that reward stylistic similarity. section 4.3.3

