liveBook · Manning

Fine-tuning

As explained in How Large Language Models Work

Fine-tuning is an additional training stage that adapts a pretrained model to a particular task or behavior using a smaller, task-relevant dataset.

Fine-tuning is a second training stage that adapts a pretrained language model to a particular task, domain, or behavior. It continues training from a base model with a smaller, specialized dataset, changing the parameters that guide its outputs. Pretraining supplies broad knowledge; fine-tuning makes that knowledge more useful for instruction following or a specific application. section 5.0 section 5.1

Fine-tuning is usually applied through two related approaches:

  • Supervised fine-tuning (SFT) repeats the ordinary training process on high-quality, task-relevant examples, such as internal manuals, medical records, legal transcripts, or code. section 5.2.1
  • Reinforcement learning from human feedback (RLHF) uses quality objectives and feedback to encourage harder-to-specify behavior, such as being a useful chatbot. section 5.2 section 5.3.3
  • SFT and RLHF can be combined in sequence, and repeated rounds can specialize a general language model for code or mathematics. section 5.2.3 section 6.2

The practical trade-off is straightforward: fine-tuning can be more effective than endlessly refining prompts, but it requires suitable data, collection effort, and hardware. It is not a guarantee of reliable behavior. A model can reproduce sensitive training information, and RLHF can optimize a reward while producing undesirable or strange results. Do not fine-tune on information that must remain private. section 5.2 section 5.2.1 section 5.3.3

A car-sales assistant

A general model may know about cars but not a company’s products, sales scripts, or internal manuals. Fine-tuning on manuals, chat histories, emails, marketing material, and carefully written examples can adapt the model to that company’s sales task. The resulting model may still need prompting, output checks, and monitoring. section 5.1 section 5.2.1 section 5.4

Figure 5.3 Supervised fine-tuning (SFT) is a simple approach to improving model results. You repeat the same process used to build the base model. Once the base model is trained on a large amount of general data, you continue training on the smaller specialized data collection.
figure

Shows the central workflow: continue training a broadly trained base model on a smaller specialized collection. section 5.2.1

How How Large Language Models Work covers it

How deep it goes
Explains SFT mechanics and RLHF objectives, but stops short of detailed optimization, implementation, and adversarial-machine-learning treatment.
Examples
Uses car sales, medical records, legal transcripts, repair manuals, code repositories, coding questions, and symbolic mathematics.
What it assumes
Assumes familiarity with pretrained or base models and training, while explaining the two-stage rationale and core methods from a practical starting point.
What you can do after
Apply it via SFT or RLHF to adapt a pretrained model, while selecting task data and checking privacy, reliability, and output risks.
How it connects
Connects pretraining, prompting, supervised fine-tuning, reinforcement learning from human feedback, retrieval augmented generation, and output validation.
Cover of How Large Language Models Work
liveBookmanning.com
Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

Compare how 37 books explain fine-tuning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.