Chapter 5: Fine-tuning LLMs for improved performance

Introduction

The introduction explains why specialized tasks may call for fine-tuning and defines it in contrast to prompting and RAG.

• Fine-tuning for specialized tasks: Explain why general-purpose LLMs may fall short in specialized domains and describe fine-tuning as further training a pretrained model on custom data to adapt its behavior.
  Why it matters: This distinction helps you recognize the kind of problem fine-tuning is meant to address.

Key terms: fine-tuning, pretrained LLM, custom data, prompting, retrieval-augmented generation (RAG)

5.1 Choosing the right approach: Prompting, RAG, or fine-tuning

Introduces fine-tuning as training a model on task-specific examples and contrasts its strength with prompting and RAG.

• Fine-tuning as model specialization: Explain that fine-tuning further trains an LLM on domain- or task-specific examples, updating its weights so it can better handle an application's nuances.
  Why it matters: This distinction helps you recognize what fine-tuning changes that a better prompt does not.
• Distinct strengths of adaptation approaches: Distinguish the strengths this section assigns to prompting, RAG, and fine-tuning without treating them as interchangeable.
  Why it matters: Each approach offers a different advantage when adapting an LLM to a custom task.

Key terms: fine-tuning, model’s weights, domain- or task-specific data, prompting, retrieval-augmented generation (RAG)

5.1.1 When to use each approach: A practical decision framework

This section gives a three-step decision framework for choosing prompting, RAG, or fine-tuning based on test results, data freshness, and consistency needs.

• Prompting as the first test: Apply the prompting test before choosing a more involved approach: improve the prompt, try it on 10 real examples, and stop if at least 8 meet the quality bar.
  Why it matters: Testing a better prompt first can avoid unnecessary development work.
• Fresh facts favor RAG: Explain why a failed prompting test followed by a need for frequently changing facts points to RAG rather than fine-tuning.
  Why it matters: Choosing retrieval for changing facts helps prevent confidently outdated answers.
• Stable consistency favors fine-tuning: Explain when to choose fine-tuning after prompting fails and the relevant data is stable, using the section's style, format, or specialized-domain examples.
  Why it matters: This distinguishes a persistent behavior problem from a fresh-information problem.

Key terms: prompting test, quality bar, real examples, system prompt, data-freshness question, RAG, fine-tuning, fresh data, consistency challenge, specialized domain knowledge

5.1.2 Hybrid approach: Combining RAG and fine-tuning

This section explains how fine-tuning and RAG complement each other, especially when an application needs both consistent behavior and current, verifiable information.

• Complementary roles of fine-tuning and RAG: Explain how a financial-advice bot can use fine-tuning to follow planning principles and risk frameworks while using RAG to retrieve current interest rates or tax regulations.
  Why it matters: The two approaches address different needs, so a production system need not choose only one.
• Source attribution limit of fine-tuning: Explain why a fine-tuned model alone cannot cite specific source documents for knowledge baked into its weights, and state what the section recommends when document references are required.
  Why it matters: A model's learned knowledge does not by itself provide the document references needed to verify an answer.

Key terms: fine-tuning, RAG, hybrid model, source attribution, verifiable answers, document references, agent tools

5.2 Case studies: Fine-tuning in the real world

5.2.1 Medicine: Med-PaLM 2

The Med-PaLM 2 case shows how medical fine-tuning can improve exam performance and alignment with medical standards, while warning that benchmark success does not guarantee reliability in practice.

• Medical fine-tuning outcomes: Explain how Google fine-tuned PaLM 2 for medicine and what the case illustrates about performance and caution in a safety-critical domain.
  Why it matters: The case connects task-specific training to both measurable performance and safer behavior in a high-stakes setting.
• Limits of medical benchmarks: Explain why a strong medical-licensing exam result does not establish real-world reliability.
  Why it matters: The distinction prevents an impressive test score from being mistaken for proof that a medical model is dependable in practice.

Key terms: fine-tuning, PaLM 2, medical standards, appropriate caution, benchmark performance, real-world reliability, novel cases, brittle

5.2.2 Finance: BloombergGPT and FinGPT

Financial-domain models can outperform general-purpose models on finance tasks, but domain training can erode general knowledge if the data is not carefully mixed.

• Financial domain specialization: Explain why financial language benefits from domain specialization, using BloombergGPT and FinGPT as examples of improved performance on financial tasks.
  Why it matters: The examples show why a model adapted to finance may be more useful for a financial task than a larger general-purpose model.
• Catastrophic forgetting risk: Explain how extensive training on financial terminology can cause catastrophic forgetting and what kind of data mixing the section says can help avoid it.
  Why it matters: Improving finance performance should not come at the unintended cost of losing general knowledge.

Key terms: BloombergGPT, FinGPT, domain specialization, market sentiment analysis, catastrophic forgetting, general-purpose examples

5.2.3 Code generation: Codex

The Codex example shows how fine-tuning can adapt a model to code, while highlighting the need for human review and high-quality training data.

• Code Fine-Tuning Capabilities and Limits: Explain how fine-tuning GPT on source code enabled programming assistance and why generated code still needs human review for complex logic.
  Why it matters: Code assistance is valuable, but its strengths do not make every generated solution reliable.
• Training Data Quality: State why training data matters to the results of fine-tuned systems such as Codex.
  Why it matters: A promising fine-tuning application still depends heavily on the data used to train it.

Key terms: fine-tuning, source code, pattern matching, boilerplate generation, human review, training data, quality, preparation

5.3 Understanding the fine-tuning process

This introduction previews the three phases of fine-tuning—data preparation, model selection, and training—before they are explained in later sections.

5.3.1 Data preparation

This section explains how to collect, curate, format, and validate fine-tuning data so a model learns the intended behavior rather than errors or accidental patterns.

• Task-matched training examples: Describe how to gather examples that resemble the inputs and desired outputs of the deployed task, and when human annotation is needed.
  Why it matters: The model can learn the behavior you want only if its examples resemble the work it will actually do.
• Data quality and coverage: Explain why examples need accuracy, appropriate coverage, and attention to sensitive information and bias.
  Why it matters: Careful curation reduces the chance that fine-tuning reinforces mistakes, harmful biases, or overly narrow behavior.
• Prompt-completion training pairs: Describe how generative fine-tuning examples pair an input prompt with the expected completion and how role or context cues clarify the intended response.
  Why it matters: Clear input-and-answer examples tell the model what response to produce for a given request.
• Consistent output examples: Explain how consistent prompt and output formatting affects the behavior learned from fine-tuning data, including when multiple output styles are wanted.
  Why it matters: Consistent examples help the fine-tuned model produce the format and style the application expects.
• Validation without data leakage: Explain why to reserve representative, separate validation examples and how their performance helps detect overfitting.
  Why it matters: A genuinely separate validation set gives a more useful signal about whether the model will handle new inputs.
• Length limits and preprocessing: Describe the practical checks needed to keep examples within token limits and remove confusing artifacts without discarding meaningful domain notation.
  Why it matters: These checks prevent unusable or confusing examples while retaining information the model needs for the domain.

Key terms: task-specific text, annotation, training data, quality over quantity, bias, balance and coverage, prompt-completion paradigm, prompt, completion, role labels

5.3.2 Fine-tuning closed-source models (OpenAI)

This section explains why and how to fine-tune an OpenAI model with custom chat data, from uploading examples to using and monitoring the resulting model.

• Fine-tuning route trade-offs: Distinguish a managed fine-tuning API from fine-tuning an open source model yourself, including when each route is useful.
  Why it matters: The route determines who manages training and which models you can fine-tune.
• Custom behavior through fine-tuning: Explain what fine-tuning is meant to improve for the e-commerce chatbot in this section.
  Why it matters: It connects the training workflow to the consistent behavior the application needs.
• OpenAI fine-tuning workflow: Trace the described workflow from preparing OpenAI training examples through starting a job and using the completed model.
  Why it matters: This is the sequence that turns custom conversation examples into a model the application can call.
• Job progress and checkpoints: Describe what can be monitored during an OpenAI fine-tuning job and why earlier checkpoints may be useful.
  Why it matters: A completed job is not necessarily the best version; progress and checkpoints provide ways to inspect training.

Key terms: cloud API service, open source model, training infrastructure, branded voice, return policy, fine-tuning, chat format, .jsonl, fine-tuning job, fine_tuned_model

5.3.3 Understanding key fine-tuning approaches

This section compares full fine-tuning, LoRA, and QLoRA by how they adapt a model, the resources they require, and their risk of catastrophic forgetting.

• Full fine-tuning tradeoffs: Explain that full fine-tuning updates every pretrained parameter, allowing comprehensive task adaptation but requiring substantial memory and compute and potentially causing catastrophic forgetting.
  Why it matters: This establishes the performance and resource tradeoff that motivates parameter-efficient alternatives.
• LoRA adapter training: Describe how LoRA freezes the base model and trains small added adapters instead of updating all its weights, reducing training resources while retaining much of full fine-tuning’s performance.
  Why it matters: LoRA makes adapting large models more practical when training the entire model is too costly.
• QLoRA memory-saving approach: Explain how QLoRA builds on LoRA by storing model weights in 4-bit form while using higher-precision calculations during training, and why this makes it attractive for limited hardware.
  Why it matters: QLoRA can make fine-tuning a powerful model feasible on more limited hardware.
• Catastrophic forgetting caveat: Distinguish the forgetting risk of full fine-tuning from the section’s warning that LoRA and QLoRA can sometimes have even higher risk, and describe the suggested training-data mitigation.
  Why it matters: A resource-efficient fine-tuning method still needs attention to preserving the model’s general abilities.

Key terms: full fine-tuning, catastrophic forgetting, Parameter-Efficient Fine-Tuning (PEFT), LoRA, base model, adapters, QLoRA, 4-bit quantization, float16, reminder examples

5.3.4 Building a customer-support assistant

This section builds a customer-support assistant with QLoRA and explains the data, model, adapter, training, and testing choices that make the example practical on consumer hardware.

• Support data and instruction format: Explain why the example uses focused customer-support interactions and how they must be formatted for LLaMA-2 instruction fine-tuning.
  Why it matters: Relevant examples help teach the desired support behavior, but the model must also receive them in the format it expects.
• Quantized base model choice: Explain why the project chooses LLaMA-2-7B and loads it with QLoRA’s 4-bit quantization rather than fine-tuning a previously saved pre-quantized model.
  Why it matters: The distinction preserves the memory advantage of QLoRA without treating a previously rounded model as the training starting point.
• LoRA adapter tradeoffs: Describe how the example’s LoRA settings control adaptation capacity, update strength, target parameters, and generalization while keeping few parameters trainable.
  Why it matters: These settings explain how the assistant can adapt substantially without the cost of updating billions of base-model parameters.
• Four-step training loop: Trace how a customer question and its correct response drive a training update.
  Why it matters: The training settings make more sense when each update is understood as a response to measured prediction error.
• Memory-conscious training configuration: Explain how batch size, gradient accumulation, learning rate, mixed precision, and training duration are chosen for this LoRA example.
  Why it matters: The configuration shows how the project balances adaptation against memory limits and overfitting risk.
• Fine-tuned assistant comparison: Describe the improvements the section expects when comparing the fine-tuned assistant with the base model, without treating them as guaranteed results.
  Why it matters: A comparison and validation data help distinguish hoped-for specialization from an improvement actually observed in testing.

Key terms: Bitext dataset, instruction-response format, [INST], [/INST], LLaMA-2-7B, QLoRA, 4-bit quantization, float16, pre-quantized, LoRA

5.4 Knowledge distillation for LLMs

The section introduces knowledge distillation as a way to retain much of a powerful LLM’s behavior while making production use faster and cheaper.

• Knowledge distillation for production: Explain why a capable model may be impractical to serve at scale and how its responses can be used to fine-tune a smaller model.
  Why it matters: It offers a way to make useful model behavior more practical to serve in production.

Key terms: knowledge distillation, teacher model, student model, training data

5.4.1 How knowledge distillation works in LLMs

This section explains how a smaller LLM can learn from a teacher’s responses and how to evaluate the resulting student for deployment.

• Sequence-level distillation mechanism: Explain how sequence-level distillation trains a smaller student model from a teacher’s final responses, without access to the teacher’s internal probabilities.
  Why it matters: You can distill a model even when the teacher exposes only its responses.
• Deployment benefits of distillation: Describe why deploying a distilled student can be more practical than deploying its teacher, in terms of cost, latency, and hardware flexibility.
  Why it matters: These trade-offs can make a resource-intensive prototype practical to serve to users.
• Student training and evaluation workflow: Trace the OpenAI distillation workflow from capturing teacher completions through benchmarking, student fine-tuning, and evaluating whether to deploy or gather more examples.
  Why it matters: A benchmark lets you check whether the smaller model performs well enough before deploying it.

Key terms: knowledge distillation, teacher model, student model, sequence-level distillation, input–output pairs, cost, latency, flexibility, commodity hardware, inference cost

5.4.2 Best practices

This section gives practical guidance for knowledge distillation and explains why a smaller student model can make deployment more efficient.

• Distillation quality practices: Describe how to choose training examples, check the student’s outputs, and keep the student current as needs change.
  Why it matters: These practices help the smaller model remain useful for the inputs and quality requirements it will face.
• Distillation deployment benefits: Explain why training a smaller student to mimic a teacher can make a high-quality LLM more practical to deploy.
  Why it matters: It connects the teacher–student approach to the practical demands of production deployment.

Key terms: teacher, student, diverse examples, evaluate frequently, retrain, knowledge distillation, speed, efficiency, scalability

5.5 Summary