How can a general-purpose language model learn to classify product reviews or follow instructions more reliably? Fine-tuning continues training a pretrained model for a particular task or behavior. It builds on what the model has already learned rather than starting from scratch. section 1.2 section 2.9
Pretraining uses large amounts of data to develop a general-purpose model. In supervised fine-tuning (SFT), a common form of post-training, the model learns from curated pairs of prompts and desirable responses. It is trained to produce responses like those examples. Instruction tuning is SFT aimed at improving instruction-following; both pretraining and SFT use next-token prediction, but SFT steers the output toward selected responses. section 1.2 section 2.9
Rating product reviews
A dataset can pair each product review with a response rating it positive, negative, or neutral. Fine-tuning on these examples aims to improve the model’s ratings. The quality of those example responses matters because the model learns to imitate them. section 2.9
Fine-tuning can also shape safer responses: an SFT dataset might show how to respond to unsafe requests. Providers also use approaches such as reinforcement learning from human feedback and AI feedback. What counts as a safe response can differ between providers, and obtaining human safety labels can impose serious financial and emotional costs on workers. section 3.8
Other uses include coding assistance and tool use. OpenAI’s Codex was fine-tuned for coding tasks. Toolformer was fine-tuned on examples containing application programming interface (API) calls and their results so it could learn when to produce tool calls. Fine-tuning is not the only route: some agents instead use examples in a prompt without additional training. section 1.7 section 8.11
