liveBook · Manning

Fine-tuning

As explained in Introduction to Generative AI, Second Edition

Fine-tuning is adapting a pretrained model through continued training for a particular task or behavior.

A general-purpose language model may need extra training to classify product reviews or follow instructions reliably. Fine-tuning adapts a pretrained model through further training, using what it has already learned rather than building a new model from scratch. section 1.2 section 2.9

Pretraining gives a model broad experience with large amounts of data. Fine-tuning then steers it toward a particular task or behavior. This approach helped address a bottleneck in earlier natural language processing: models built for individual tasks often required large amounts of expensive, time-consuming annotated data. section 1.2

A common approach is supervised fine-tuning (SFT), in which examples provide responses the model should learn to imitate. The examples are selected so that responses match the desired outputs, not merely sound like natural language. A relatively small amount of high-quality data can be effective. section 2.9

Rating product reviews

To train a model to classify sentiment, examples can pair product reviews with ratings of positive, negative, or neutral. The model is then trained to produce responses resembling those ratings. Instruction tuning uses similar prompt-and-response pairs to improve instruction-following more generally. section 2.9

Fine-tuning can also shape safety behavior or teach tool use. For safety, examples may pair unsafe inputs with appropriate responses, although providers can disagree about what counts as safe. In Toolformer, a model was fine-tuned on examples containing tool calls and their results, helping it learn when calls were useful. Fine-tuning is not the only route to tool use: some systems instead provide examples in a prompt. section 3.8 section 8.11

Training data and the people who prepare it matter. Safety work can expose labelers to disturbing content, and outsourced workers may lack adequate pay or protections. Fine-tuning is also not a guarantee against the broader risks of language models, including biased or invented outputs. section 3.8 section 1.23

How Introduction to Generative AI, Second Edition covers it

How deep it goes
Explains supervised examples and next-token prediction at an overview level, without a training procedure or implementation details.
Examples
Uses product-review sentiment ratings, safe responses to unsafe inputs, and Toolformer’s API calls as examples.
What you can do after
Evaluate and understand how fine-tuning steers a pretrained model and why example quality and labor conditions matter.
How it connects
Connects pretraining with supervised learning, instruction tuning, reinforcement learning, and few-shot prompting.

Also explained in

Compare how 52 books explain fine-tuning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Introduction to Generative AI, Second Edition. The section links point to the parts of the book it draws on.