4 Instruction fine-tuning
This chapter covers
- Why instruction fine-tuning is the foundation
before RLHF - How models are fine-tuned today to follow
instructions
Early large pretrained language models were trained with a next-token prediction objective and, by default, did not come with an explicit interface for following instructions. Around the release of GPT-3 [1], prompting and in-context learning became a widely used way to adapt a single model to many tasks (although task-specific fine-tuning remained common) by showing examples in context and asking the model to complete a similar task. A practical next step was instruction fine-tuning, which teaches the model to respond in an instruction–response format rather than just continuing text. For example, given the input “What is the boiling point of water?” a base language model might continue with related but unhelpful text:
An instruction-tuned model, by contrast, answers the question directly: