liveBook · Manning

Fine-tuning

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Fine-tuning is the additional training of a pretrained model on a task-specific or domain-specific dataset.

A pretrained language model can complete text without reliably following an instruction. Fine-tuning gives it additional training on curated examples of the behavior needed for a task or domain. In supervised fine-tuning (SFT), each example pairs a prompt with a demonstration response. The objective remains next-token prediction, but the training data changes from raw text to instruction-response pairs. section 7.2.2

An earnings-report assistant

A base model asked to summarize a quarterly earnings report might produce a generic template instead of an answer. SFT examples can pair similar requests with short, structured summaries that lead with the headline number. Training on such demonstrations teaches the model the desired response pattern. section 7.2.2

Demonstrations can be single-turn answers, prompts containing several examples, step-by-step responses, or multi-turn tool-use traces. The resulting model can follow instructions and match formats represented in its training data, but it primarily learns to imitate those demonstrations. High-quality examples alone do not provide a direct penalty for bad behaviors absent from the dataset. section 7.2.2

A practical alignment workflow using SFT:

  • Write a concise behavioral spec and collect prompts from the production distribution, emphasizing known failures. section 8.5.2
  • Use a critic model to critique and revise responses against the spec, then fine-tune the production model on the prompt and revised-response pairs. section 8.5.2
  • Test the fine-tuned model against the spec, inspect failures, and add them to the next training iteration. Preference-based reinforcement learning is an optional later step when the necessary infrastructure is available. section 8.5.2

Data quality determines what the model learns to imitate. In one SFT pipeline, labelers wrote responses under a rubric for helpfulness, honesty, and harmlessness; unclear rubrics or inconsistent labelers are diagnostic starting points when results disappoint. Alternatively, training on responses verified against authoritative sources can teach source faithfulness and uncertainty calibration. Continue monitoring: no mitigation eliminates hallucination. section 7.5.2 section 5.4.5

Figure 7.8 The InstructGPT three-stage pipeline. Stage 1 fine-tunes the pretrained model on labeler-written demonstrations. Stage 2 trains a reward model on labeler-ranked comparisons of model outputs. Stage 3 optimizes the SFT model with reinforcement learning using PPO, with the reward model's score (penalized by KL divergence from the SFT model) as the reward signal. The SFT model from stage 1 plays two roles: it initializes the RL policy and is the reference distribution in the KL penalty. Source: Figure 2 of Ouyang et al. 2022.

The pipeline shows SFT as the starting point for preference training: its model initializes the reinforcement-learning policy and serves as the reference for a divergence penalty. section 7.5.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains training objectives, demonstration formats, data preparation, iteration, and imitation limits without detailing a fine-tuning implementation.
Examples
Illustrates SFT with earnings-report summaries, labeler-written demonstrations, and responses revised by an AI critic.
What you can do after
Evaluate how demonstration quality and coverage affect fine-tuning, and plan a critique-revise, train, and test cycle.
How it connects
Connects fine-tuning to pretraining, next-token prediction, reinforcement learning from human feedback, reward modeling, and constitutional AI.

Also explained in

Compare how 52 books explain fine-tuning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.