liveBook · Manning

Fine-tuning

As explained in Fight Fraud with Machine Learning

Fine-tuning is adapting a pretrained machine-learning model by training it further on task-specific data.

Fine-tuning adapts a pretrained GPT model through further supervised training on labeled task data. For conversational behavior, the data pairs questions as inputs with manually labeled answers as outputs, teaching the model to answer prompts rather than simply continue text. section 5.2.1

In the illustrated ChatGPT training sequence:

  • Pretraining learns from a large text corpus without output labels and consumes the most compute and training time. section 5.2.1
  • Supervised fine-tuning uses labeled question-and-answer pairs to shape responses to prompts. section 5.2.1
  • Reinforcement learning further tunes the supervised model using a reward model trained from human rankings or scores. section 5.2.1
Figure 5.12 Demonstration of Supervised Fine-Tuning of a ChatGPT model.

Compare unlabeled pretraining data with the paired inputs and outputs used for supervised fine-tuning. section 5.2.1

How Fight Fraud with Machine Learning covers it

How deep it goes
Explains the supervised data format and its contrast with pretraining, then defers further improvement to reinforcement learning.
Examples
Uses manually labeled question-and-answer pairs and a diagram contrasting unsupervised text data with supervised pairs.
What you can do after
Evaluate/understand why labeled answers make a pretrained GPT respond to prompts instead of only continuing text.
How it connects
Connects fine-tuning to GPT, pretraining, supervised learning, reinforcement learning, reward models, and RLHF.

Also explained in

Compare how 52 books explain fine-tuning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fight Fraud with Machine Learning. The section links point to the parts of the book it draws on.