liveBook · Manning

Superficial alignment hypothesis

As explained in Reinforcement Learning from Human Feedback

The superficial alignment hypothesis is the claim that pretraining supplies most knowledge and capability while alignment mainly teaches a model a narrow set of interaction formats or styles.

The superficial alignment hypothesis claims that pretraining supplies nearly all knowledge and capability, while alignment mainly teaches which interaction formats or styles a model should use. section 1.4

The hypothesis implies a particular view of post-training:

  • A pretrained model already contains broad capabilities, but expresses them through next-token prediction rather than useful question-answering. section 1.4
  • A small instruction-tuning dataset may substantially change behavior and improve narrow evaluations, without unlocking more difficult capabilities. section 1.4
  • The hypothesis therefore treats alignment as largely stylistic, but broader performance may depend strongly on data scale and selection. section 1.4

Small-data alignment

A few thousand instruction-tuning examples can improve AlpacaEval, MT Bench, or Arena results, yet those gains need not transfer to challenging capabilities. section 1.4

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Defines the hypothesis, derives its small-data corollary, and stops short of endorsing narrow benchmark gains as broad capability evidence.
Examples
Uses a few thousand instruction examples and AlpacaEval, MT Bench, and Arena results to illustrate narrow improvement.
What you can do after
Evaluate whether post-training gains reflect interaction-style changes or broader capability improvements.
How it connects
Connects pretraining, alignment, instruction tuning, post-training, elicitation theory, and capability evaluation.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.