liveBook · Manning

Supervised fine-tuning

As explained in Machines that Think

Supervised fine-tuning adjusts a pretrained model using labeled examples of desired inputs and outputs or behaviors.

Supervised fine-tuning (SFT) shapes a pretrained model using examples of desired behavior. Pretraining gives a language model broad competence at predicting text; SFT makes particular responses more likely when an application needs it to follow instructions, adopt a role, or maintain an appropriate tone. section 7.1.1

Training examples are usually instruction–response pairs: a request followed by an answer written, selected, or curated by human annotators. The model predicts the answer token by token. Differences between its predictions and the target answer produce errors that update its parameters, making similar responses more likely in future. section 7.1.1

Figure 7.1 Supervised fine-tuning (SFT). The model is trained on curated instruction–response examples that represent desired behavior. During training, the model predicts the response token by token and compares its predictions with the target sequence. Errors adjust the model’s parameters so that responses resembling the reference answers become more likely in future generations.

Follow how curated examples, token predictions, and parameter updates shape future responses. section 7.1.1

The examples can shape a general-purpose chatbot, coding helper, customer-support agent, tutor, or domain-specific assistant. They can also teach responses to harmful requests, such as refusing, setting boundaries, or redirecting an interaction toward a safer response. These behaviors are learned tendencies, not fixed rules or evidence of an intrinsic ethical commitment. section 7.1.1 section 7.5

Imitating a selected answer does not resolve every choice: multiple responses may satisfy an instruction while differing in helpfulness, honesty, caution, tone, or safety. Later preference-based alignment can refine behavior by comparing alternatives or applying guiding principles. SFT can also begin the shaping of reasoning models with examples that solve problems through useful intermediate steps, followed by reward-based training. section 7.1.2 section 9.2.2

How Machines that Think covers it

How deep it goes
Explains instruction–response training through token prediction, target comparison, and parameter updates, then distinguishes preference-based alignment.
Examples
Uses chatbot, coding helper, customer-support agent, and tutor roles, along with refusals and intermediate-step problem solving, to illustrate desired behavior.
What you can do after
Evaluate or understand how curated examples change response tendencies without guaranteeing ethical behavior.
How it connects
Connects pretraining and alignment with preference learning and reasoning models.

Also explained in

Compare how 3 books explain supervised fine-tuning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.