Chapter 4: Instruction fine-tuning

Introduction

Introduces instruction fine-tuning as the supervised step that makes pretrained language models follow instructions and prepares them for RLHF.

• Instruction Following Versus Continuation: Distinguish how a base language model and an instruction-tuned model might respond to a question, and explain what instruction fine-tuning teaches the model to do.
  Why it matters: This distinction explains why a capable pretrained model may still need training to act like an assistant.
• Converging Paths to Instruction Tuning: Explain the two developments that made instruction fine-tuning practical and useful across many tasks.
  Why it matters: These developments explain why instruction tuning became a general approach rather than a separate setup for each task.
• Instruction Tuning as RLHF Foundation: Explain why instruction fine-tuning precedes RLHF and why training-data structure matters to later stages.
  Why it matters: RLHF needs a model that can respond to instructions before its responses can be used in later training.

Key terms: next-token prediction, instruction fine-tuning, instruction–response format, text-to-text, prompting, in-context learning, instruction–response examples, instruction fine-tuning (IFT), supervised fine-tuning (SFT), preference data collection

4.1 Chat templates and the structure of instructions

This section explains how chat templates turn role-labeled conversations into token sequences that instruction-tuned language models can continue.

• Base prompting and chat formatting: Distinguish prompting a pretrained base model with tokens to continue from formatting a conversation with a chat template for post-training.
  Why it matters: It explains why post-training examples need more structure than a plain prompt for continuation.
• Conversation message roles: Describe what the system, user, and assistant roles identify, including the placement and purpose of a system message.
  Why it matters: Roles tell the model which text is instruction or context, which is the user's message, and which is an assistant response.
• Template to token sequence: Trace how the described chat template turns role-labeled messages into model input and cues the assistant to generate a response.
  Why it matters: The final assistant tag makes a formatted conversation a prompt the model can answer.
• Shared format and template variants: Explain why training data is packed into a consistent chat format while recognizing that different chat templates exist.
  Why it matters: A consistent format carries conversations through training without implying that every model uses the same template.

Key terms: pretrained language model, chat template, post-training, system, user, assistant, system prompts, chat markup, <|im_start|>assistant, <|im_end|>

4.2 Best practices for instruction tuning

This section identifies the data choices that matter most in instruction tuning and explains why they should be judged in the context of the full post-training process.

• Routes to effective instruction tuning: Describe how effective instruction tuning can use accessible fine-tuning methods or small, focused datasets for narrow domains, while noting the shift toward large synthetic datasets for most tasks.
  Why it matters: It keeps you from treating one dataset size or training method as a universal recipe.
• Completion quality as training signal: Explain why high-quality completions are especially important in instruction-tuning data.
  Why it matters: It directs attention to the responses supplying the model's learning signal.
• Prompt selection and overall optimization: Explain how prompt relevance and diminishing returns affect dataset design, and why later training stages matter when judging instruction-tuning data.
  Why it matters: It helps you prioritize relevant prompts and the full training process over sheer dataset size or perfection at one stage.

Key terms: instruction tuning, quantization, small, focused datasets, synthetic datasets, high-quality data, completions, prompts, downstream tasks, diminishing returns, overall optimization

4.3 Implementation details

This section explains how instruction fine-tuning keeps pretraining’s autoregressive loss while changing batch sizes, masking, and learning rates.

• Smaller post-training batches: Explain why instruction tuning uses smaller batches than pretraining and how that affects the number of GPUs used by a training job.
  Why it matters: Batch size changes both the optimization setting and how much hardware a single SFT job can use.
• Response-focused loss masking: Distinguish instruction tuning’s loss application from pretraining’s, despite their use of the same autoregressive loss function.
  Why it matters: The masking determines which part of an instruction example the model is directly trained to produce.
• Multiturn assistant masking choices: Describe the two masking choices for multiturn conversations and what each choice trains on.
  Why it matters: These choices decide whether earlier assistant replies contribute directly to the training loss.
• Conservative SFT learning rates: Explain why SFT typically uses a lower learning rate than pretraining and why sequence packing can support a higher SFT rate.
  Why it matters: Learning-rate choices need to account for both the pretrained starting point and how many useful tokens contribute to an update.

Key terms: batch size, global batch size, per-device batch size, SFT, autoregressive loss, prompt masking, prompt tokens, responses, multiturn masking, final-turn only

4.4 Summary