19 Style and information
Early developments in RLHF gave it a reputation for being "just style transfer" or other harsh critiques on how RLHF manipulates the way information is presented in outputs.
Style transfer has held back the RLHF narrative for two reasons.
First, when people discuss style transfer, they don’t describe this as being important or exciting. Style is a never-ending source of human value, it’s why retelling stories can result in new bestselling books (such as Sapiens: A Brief History of Humankind1), and it is a fundamental part of continuing to progress our intellectual ecosystem. Style is intertwined with what the information is.
Second, we’ve seen how different styles actually can improve evaluation improvements with Llama 3 [1]. The Llama 3 Instruct models scored extremely high on ChatBotArena, and it’s accepted as being because they had a more fun personality. If RLHF is going to make language models simply more fun, that is delivered value.
Throughout this chapter, the term "chattiness" is used to encompass the growing length of responses from models training with RLHF, but it also encompasses techniques like heavy markdown use, emojis, and formatting the answer in bulleted lists.