Supervised fine-tuning adapts a language model using curated examples of inputs paired with desired responses. It is imitation learning: the model learns to reproduce demonstrations, rather than explore responses and receive rewards. SFT teaches baseline behavior, format, and tone, but does not teach the model to prefer one acceptable response over another. section 10.5 section 10.9
A practical SFT stage:
- Prepare high-quality query-response pairs that demonstrate the desired behavior. section 10.8.3
- Use the prompt as context, but mask its labels so loss is computed only on assistant response tokens. section 10.8.3
- Train and validate the model; the demonstrated implementation uses AdamW, a warm-up followed by cosine learning-rate decay, and skips batches with NaN loss. section 10.8.3
Medical patient-support chatbot
Demonstrations can teach a chatbot to respond empathetically, avoid giving diagnoses, recommend professional care for serious or uncertain symptoms, and escalate emergencies. SFT establishes these behaviors before reward-model training and PPO. section 10.8 section 10.8.3
Listing 10.1 SFT Dataset with Label Masking
class MedicalSFTDataset(Dataset):
def __init__(
self,
data: List[Dict],
tokenizer: PreTrainedTokenizer,
max_length: int = 512,
):
self.data = data
self.tokenizer = tokenizer
self.max_length = max_length
def __len__(self) -> int:
return len(self.data)
def __getitem__(self, idx: int) -> Dict[str, torch.Tensor]:
item = self.data[idx]
prompt = self._format_prompt(item["query"]) #A
full_text = prompt + item["response"]
➥ + self.tokenizer.eos_token #B
encodings = self.tokenizer(
full_text,
truncation=True,
max_length=self.max_length,
padding="max_length",
return_tensors="pt",
)
prompt_encodings = self.tokenizer(
prompt,
truncation=True,
max_length=self.max_length,
return_tensors="pt",
)
prompt_length = prompt_encodings["input_ids"]
➥.shape[1] #C
labels = encodings["input_ids"].clone()
labels[0, :prompt_length] = -100 #D
labels[labels == self.tokenizer
➥.pad_token_id] = -100 #E
return {
"input_ids": encodings["input_ids"].squeeze(0),
"attention_mask": encodings["attention_mask"].squeeze(0),
"labels": labels.squeeze(0),
}
def _format_prompt(self, query: str) -> str:
return (
"You are a helpful…"
f"Patient: {query}\n\n"
"Assistant: "
) #FThe dataset listing shows how prompt-response examples are tokenized and prompt labels are masked. section 10.8.3


