Fine-tuning continues training from a pretrained language model rather than learning language from random initialization. It adapts an already capable model toward a narrower task or behavior, such as answering instructions directly, following a tool-use format, or reflecting preference signals. In post-training, instruction fine-tuning (also called supervised fine-tuning, or SFT) establishes the instruction–response interface that later preference optimization and RL build on. section 3.1.4 section 4.0 section 3.1.5
For instruction fine-tuning, training retains the next-token loss but usually masks prompt tokens so the loss teaches responses rather than user queries. Multiturn data can similarly select which assistant turns contribute to loss. Smaller batches than pretraining help optimize on the narrower distribution while retaining pretrained generalization; high-quality completions and prompts resembling downstream tasks are central practical choices. section 4.3 section 4.2 section 4.4
From continuation to answer
Given “What is the boiling point of water?”, a base model may continue with related chemistry questions. An instruction-tuned model instead returns a direct answer: 100C (212F) at standard atmospheric pressure. Fine-tuning changes the expected interaction format, not merely the factual content available to the model. section 4.0
Fine-tuning appears at several points in a modern post-training pipeline:
- Instruction fine-tuning imitates selected instruction–response examples to teach basic instruction following and formatting. section 3.1.5
- Reward-model fine-tuning trains an instruction-tuned model on pairwise preferences to produce a scalar quality signal for RL. section 1.3
- Tool-use SFT trains on trajectories to establish tool selection and structured call formatting before preference optimization or environment-feedback RL. section 13.5 section 13.6
- Rejection sampling can select preferred completions and then apply standard instruction fine-tuning to the current model. section 9.1.3
Fine-tuning is not automatically benign. SFT can improve in-distribution performance while memorizing a single-task training distribution and degrading under distribution shift or sequential training. In RLHF, optimization also needs protection against drifting too far from the pretrained policy; a KL-divergence penalty treats deviation from a reference policy as a constrained budget. Larger penalty weights preserve more of the starting policy, while smaller weights permit stronger reward chasing. section 15.2.1 section 15.2.2 section 3.1.4
Equation 3.11
\[\label{eq:rlhf_opt_eq} {\max_\pi \; \mathbb{E}_{\tau \sim \pi} \left[r_\theta(s_t, a_t)\right] - \beta \mathcal{D}_{\text{KL}}(\pi_{\text{RL}}(\cdot|s_t) \| \pi_{\text{ref}}(\cdot|s_t)).}\]This objective makes the fine-tuning trade-off explicit: maximize learned reward while penalizing divergence from the reference policy through β-weighted KL distance. section 3.1.4
Early RLHF recipes used roughly 10,000 instruction examples, then reward-model training on preference pairs, then RL optimization. More recent recipes may use about one million synthetic instruction examples and combine instruction tuning with on-policy preference data and RL with verifiable rewards. Scale helps, but returns can diminish, and the overall sequence of optimization stages matters more than treating any stage as sufficient alone. section 3.2.1 section 3.2.2 section 4.2
