A pretrained language model can complete text without reliably following an instruction. Fine-tuning gives it additional training on curated examples of the behavior needed for a task or domain. In supervised fine-tuning (SFT), each example pairs a prompt with a demonstration response. The objective remains next-token prediction, but the training data changes from raw text to instruction-response pairs. section 7.2.2
An earnings-report assistant
A base model asked to summarize a quarterly earnings report might produce a generic template instead of an answer. SFT examples can pair similar requests with short, structured summaries that lead with the headline number. Training on such demonstrations teaches the model the desired response pattern. section 7.2.2
Demonstrations can be single-turn answers, prompts containing several examples, step-by-step responses, or multi-turn tool-use traces. The resulting model can follow instructions and match formats represented in its training data, but it primarily learns to imitate those demonstrations. High-quality examples alone do not provide a direct penalty for bad behaviors absent from the dataset. section 7.2.2
A practical alignment workflow using SFT:
- Write a concise behavioral spec and collect prompts from the production distribution, emphasizing known failures. section 8.5.2
- Use a critic model to critique and revise responses against the spec, then fine-tune the production model on the prompt and revised-response pairs. section 8.5.2
- Test the fine-tuned model against the spec, inspect failures, and add them to the next training iteration. Preference-based reinforcement learning is an optional later step when the necessary infrastructure is available. section 8.5.2
Data quality determines what the model learns to imitate. In one SFT pipeline, labelers wrote responses under a rubric for helpfulness, honesty, and harmlessness; unclear rubrics or inconsistent labelers are diagnostic starting points when results disappoint. Alternatively, training on responses verified against authoritative sources can teach source faithfulness and uncertainty calibration. Continue monitoring: no mitigation eliminates hallucination. section 7.5.2 section 5.4.5
Figure 7.8 The InstructGPT three-stage pipeline. Stage 1 fine-tunes the pretrained model on labeler-written demonstrations. Stage 2 trains a reward model on labeler-ranked comparisons of model outputs. Stage 3 optimizes the SFT model with reinforcement learning using PPO, with the reward model's score (penalized by KL divergence from the SFT model) as the reward signal. The SFT model from stage 1 plays two roles: it initializes the RL policy and is the reference distribution in the KL penalty. Source: Figure 2 of Ouyang et al. 2022.
The pipeline shows SFT as the starting point for preference training: its model initializes the reinforcement-learning policy and serves as the reference for a divergence penalty. section 7.5.1



















































