14 Style and Information
This chapter covers
- Why “it just changes style” undersells RLHF’s value
- How a language model’s style affects both user experience and benchmark scores
- Balancing chattiness of models
Early developments in RLHF gave it a reputation for being “just style transfer” or other harsh critiques on how RLHF manipulates the way information is presented in outputs. This chapter explains why style is core to understanding the value RLHF provides — and why it positively impacts both model capability and user experience.
The idea of RLHF being solely about style transfer has held back the RLHF narrative for two reasons. The first is how RLHF became associated with small, unimportant changes to the model. When people discuss style transfer, they don’t describe this as being important or exciting – they think of it as superficial. Yet, style is a never-ending source of human value; it’s why retelling stories can result in new bestselling books (such as Sapiens), and it is a fundamental part of continuing to progress our intellectual ecosystem. Style is intertwined with what the information is.