liveBook · Manning

Supervised fine-tuning

As explained in Applied Reinforcement Learning

Supervised fine-tuning is the training of a pre-trained model on labeled examples so that it produces desired outputs for given inputs.

Supervised fine-tuning adapts a language model using curated examples of inputs paired with desired responses. It is imitation learning: the model learns to reproduce demonstrations, rather than explore responses and receive rewards. SFT teaches baseline behavior, format, and tone, but does not teach the model to prefer one acceptable response over another. section 10.5 section 10.9

A practical SFT stage:

  • Prepare high-quality query-response pairs that demonstrate the desired behavior. section 10.8.3
  • Use the prompt as context, but mask its labels so loss is computed only on assistant response tokens. section 10.8.3
  • Train and validate the model; the demonstrated implementation uses AdamW, a warm-up followed by cosine learning-rate decay, and skips batches with NaN loss. section 10.8.3

Medical patient-support chatbot

Demonstrations can teach a chatbot to respond empathetically, avoid giving diagnoses, recommend professional care for serious or uncertain symptoms, and escalate emergencies. SFT establishes these behaviors before reward-model training and PPO. section 10.8 section 10.8.3

Listing 10.1 SFT Dataset with Label Masking
class MedicalSFTDataset(Dataset):
    def __init__(
        self,
        data: List[Dict],
        tokenizer: PreTrainedTokenizer,
        max_length: int = 512,
    ):
        self.data = data
        self.tokenizer = tokenizer
        self.max_length = max_length
 
    def __len__(self) -> int:
        return len(self.data)
 
    def __getitem__(self, idx: int) -> Dict[str, torch.Tensor]:
        item = self.data[idx]
 
        prompt = self._format_prompt(item["query"])  #A
        full_text = prompt + item["response"]
        ➥ + self.tokenizer.eos_token  #B
 
        encodings = self.tokenizer(
            full_text,
            truncation=True,
            max_length=self.max_length,
            padding="max_length",
            return_tensors="pt",
        )
 
        prompt_encodings = self.tokenizer(
            prompt,
            truncation=True,
            max_length=self.max_length,
            return_tensors="pt",
        )
        prompt_length = prompt_encodings["input_ids"]
       ➥.shape[1]  #C
 
        labels = encodings["input_ids"].clone()
        labels[0, :prompt_length] = -100  #D
        labels[labels == self.tokenizer
        ➥.pad_token_id] = -100  #E
 
        return {
            "input_ids": encodings["input_ids"].squeeze(0),
            "attention_mask": encodings["attention_mask"].squeeze(0),
            "labels": labels.squeeze(0),
        }
 
    def _format_prompt(self, query: str) -> str:
        return (
            "You are a helpful…"
            f"Patient: {query}\n\n"
            "Assistant: "
        )  #F

The dataset listing shows how prompt-response examples are tokenized and prompt labels are masked. section 10.8.3

How Applied Reinforcement Learning covers it

How deep it goes
Details response-only label masking, dataset preparation, and training choices, while omitting standard evaluation and checkpointing boilerplate.
Examples
Uses medical chatbot demonstrations to teach empathy, appropriate safety language, and emergency escalation.
What it assumes
Assumes a pretrained, usually instruction-tuned language model; explains why starting from a raw pretrained model is uncommon.
What you can do after
Build an SFT training stage that learns from curated prompt-response pairs and computes loss only on response tokens.
How it connects
Connects imitation learning with reward modeling, reinforcement learning from human feedback, and proximal policy optimization.

Also explained in

Compare how 3 books explain supervised fine-tuning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.