17 Crafting model character and products
This chapter covers
- Shaping model personality with character training
- Mechanistic tools for personality control
- Understanding model specifications and why they matter
- The intersection of RLHF with product development
Frontiers in RLHF and post-training show how companies use these techniques to make leading products. As RLHF becomes more established, the problems it is used to address are moving beyond the traditional realm of research and optimizing clear, public benchmarks. In this chapter, we discuss a series of use cases for RLHF and post-training that are not well-established in the academic literature but are essential at leading AI laboratories, with a primary focus on the process that teaches language models their personality.
17.1 Character training
The default way for users to change a model’s behavior is to write a prompt describing the change at inference-time: for example, instead of prompting a model, “Write me an email summarizing my last month of work,” we can write “Acting as a burned out employee, write me an email summarizing my last month of work.”