Fine-tuning adapts a pretrained model to a narrower task or behavior by continuing optimization on task-specific examples. In supervised fine-tuning (SFT), labeled examples supply target responses, and training adjusts model parameters to reduce prediction error. This sits between broad pretraining and any later preference-alignment phase: SFT teaches the model the kinds of responses its role requires; alignment can further shape those responses around human preferences. section 5.2.1
A practical SFT workflow has five parts:
- Choose a pretrained model and establish a baseline by generating responses before training. section 7.4 section 7.5
- Prepare supervised examples in the format the model will see. Instruction data can use instruction, input, and response fields; chat data can use system, user, and assistant messages. section 5.2.1 section 7.4
- Configure optimization for the available compute. One Mistral-7B workflow loads a 4-bit model and adds LoRA adapters so training updates only a small portion of it; a Gemma translation workflow uses batch size, gradient accumulation, and memory-saving settings. section 5.2.1 section 7.5
- Train with TRL's SFTTrainer, then save the resulting model or checkpoint. section 7.4 section 7.5
- Compare outputs before and after training on held-out or fixed prompts. For the prosocial adaptation, a further check computes perplexity on an evaluation slice. section 7.4 section 7.5
Specializing a small model for translation
For English–Spanish translation, filter and sample OPUS-100 sentence pairs, then format each pair as a chat exchange: a system instruction specifies translation, the user supplies English, and the assistant supplies Spanish. Fine-tune Gemma 3 270M on these examples and rerun the initial translation prompts. The shown outputs improve on some baseline failures, but still include an inaccurate translation; inspect actual generations rather than treating completion of training as proof of quality. section 7.5
Listing 7.16 Running SFT with TRL on chat-formatted samples
torch_dtype = model.dtype
args = SFTConfig(
output_dir=checkpoint_dir,
max_length=512,
packing=False,
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_checkpointing=False,
optim="adamw_torch_fused",
logging_steps=10,
save_strategy="epoch",
eval_strategy="epoch",
learning_rate=learning_rate,
fp16=(torch_dtype == torch.float16),
bf16=(torch_dtype == torch.bfloat16),
lr_scheduler_type="cosine",
push_to_hub=False,
report_to="tensorboard",
dataset_kwargs={
"add_special_tokens": False,
"append_concat_token": True,
},
)
trainer = SFTTrainer(
model=model,
args=args,
train_dataset=train_built,
eval_dataset=eval_built,
processing_class=tokenizer,
)
trainer.train()
trainer.save_model()
The chat-data training listing shows how SFTConfig sets epochs, batch size, learning rate, evaluation, and checkpointing before SFTTrainer runs training. section 7.4
Fine-tuning can target behavior as well as a conventional task. A prosocial dialogue workflow builds chat examples with an empathy-oriented system message, trains Gemma 3 270M for three epochs, and compares its replies with those of the starting model. The aim is a more suitable conversational stance, not merely a different answer format. section 7.4
