liveBook · Manning

Prefix bias

As explained in Reinforcement Learning from Human Feedback

Prefix bias is a bias in which the beginning of a model completion disproportionately influences a preference judgment.

Prefix bias occurs when the beginning of a model completion disproportionately influences a preference judgment. Because preference comparisons provide the proxy signal for human values, this bias can transfer into the trained model. section 11.3 section 11.5

Treat prefix bias as a preference-data quality risk:

  • Audit comparisons for judgments driven primarily by a completion’s opening rather than its full response. section 11.3
  • Account for transfer into RLHF training, where biased preference data can shape the final model. section 11.3 section 11.5
  • Expect mitigation effectiveness to vary, since subtle biases are difficult to detect and interventions are not uniformly reliable. section 11.3

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Defines the judgment pattern and its downstream risk but does not provide a concrete detection or mitigation procedure.
What it assumes
Assumes familiarity with model completions, human preference judgments, preference data, and RLHF training.
What you can do after
Diagnose preference-data quality risks that may transfer into trained-model behavior.
How it connects
Connects prefix bias to preference data, pairwise comparisons, reward functions, RLHF, and synthetic preference data.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.