Chapter 7: Fine-tuning to follow instructions

Introduction

The chapter introduction contrasts instruction fine-tuning with the earlier classification task and explains why following instructions matters for conversational LLMs.

• Classification and instruction fine-tuning: Distinguish fine-tuning an LLM to classify text from fine-tuning it to follow human instructions, and explain why instruction fine-tuning is useful for conversational applications.
  Why it matters: This distinction clarifies why the chapter uses instruction data rather than the classification approach from chapter 6.

Key terms: fine-tuning for classification, instruction fine-tuning, instruction dataset

7.1 Introduction to instruction fine-tuning

This section explains why a pretrained text-completion LLM may need instruction fine-tuning to produce responses to specific requests.

• Text completion versus instruction following: Distinguish a pretrained LLM’s ability to complete text from its ability to follow specific instructions, and explain what instruction fine-tuning aims to improve.
  Why it matters: The distinction explains why a model that writes fluent continuations may still need further training for instruction-based tasks.

Key terms: pretrained LLM, text completion, instruction fine-tuning, supervised instruction fine-tuning

7.2 Preparing a dataset for supervised instruction fine-tuning

This section prepares an instruction–response dataset by inspecting its entries, formatting them as Alpaca-style prompts, and dividing them into training, validation, and test sets.

• Instruction dataset structure: Describe how the JSON instruction dataset is represented and what information an entry contains. Explain that instruction fine-tuning uses explicitly provided input-output pairs.
  Why it matters: The entry structure determines what can be turned into a training prompt and response.
• Alpaca prompt formatting: Explain why the section uses the Alpaca prompt style and describe how format_input handles an entry whose 'input' field is empty.
  Why it matters: Consistent prompt formatting lets entries with and without extra input be prepared for the same fine-tuning process.
• Dataset partitioning: Describe how the prepared instruction dataset is divided before data loaders are set up.
  Why it matters: The split prepares separate sets for the subsequent fine-tuning workflow.

Key terms: instruction fine-tuning, instruction, input, output, JSON, prompt styles, Alpaca prompt style, Phi-3, format_input, ### Input:

7.3 Organizing data into training batches

This section prepares instruction-fine-tuning batches by formatting and tokenizing examples, padding them, creating next-token targets, and excluding padding from the loss.

• Instruction dataset preparation: Describe how InstructionDataset prepares examples before batching: it applies format_input and pretokenizes the inputs in its constructor.
  Why it matters: It separates preparation of individual examples from the work needed to assemble a batch.
• Per-batch padding: Explain why instruction fine-tuning uses a custom collate function and how it pads tokenized examples within a batch.
  Why it matters: Examples must have a uniform length within a batch, but need not carry padding required by unrelated batches.
• Next-token target alignment: Describe how the collate function prepares target token IDs alongside input token IDs and what the shifted alignment teaches the model.
  Why it matters: The shifted targets turn each token position into a next-token training prediction.
• Padding loss exclusion: Explain why extra padding IDs are replaced with -100 in the targets while one end-of-text ID is retained.
  Why it matters: This keeps artificial padding out of the learning signal without removing the signal for when a response should end.
• Optional instruction masking: Distinguish optional instruction-target masking from the padding masking used here, including its intended effect and the section's stated uncertainty about its benefit.
  Why it matters: It clarifies that excluding artificial padding is part of this batching approach, whereas excluding instruction targets is a separate, optional choice.

Key terms: InstructionDataset, format_input, pretokenizes, __init__, custom collate function, DataLoader, padding, <|endoftext|>, 50256, input token IDs

7.4 Creating data loaders for an instruction dataset

This section connects the instruction dataset and custom collate function to PyTorch data loaders, configures batching for the model, and explains the resulting batch shapes.

• Instruction data loader setup: Describe how InstructionDataset objects and the custom_collate_fn are plugged into PyTorch data loaders so the loaders can organize instruction-fine-tuning batches.
  Why it matters: This connects the prepared examples to the batches used for fine-tuning.
• Collate function device configuration: Explain how the chosen device is supplied to custom_collate_fn and why this section transfers input and target tensors there rather than in the main training loop. State why allowed_max_length is set to 1024.
  Why it matters: The configuration prepares batches for the selected device without making device transfers block the training loop.
• Variable batch token lengths: Interpret the displayed input and target batch dimensions and explain why different batches can have different token lengths.
  Why it matters: The batch shapes show how to read loader output without assuming every batch has the same token length.

Key terms: InstructionDataset, custom_collate_fn, PyTorch data loaders, device, partial, allowed_max_length, GPT-2, input and target batch, batch size, tokens

7.5 Loading a pretrained LLM

The section loads a larger pretrained GPT model for instruction fine-tuning and checks its instruction-following performance before training.

• Medium pretrained model choice: Explain why the section loads gpt2-medium rather than gpt2-small for instruction fine-tuning and what changes in the loading code.
  Why it matters: Model capacity affects whether fine-tuning can produce useful instruction-following behavior.
• Generated response isolation: Describe why and how the section extracts just the model’s response from generated text.
  Why it matters: Separating the response from its prompt makes the model’s answer easier to inspect.
• Pre-fine-tuning instruction baseline: Explain how the section checks the pretrained model before fine-tuning and what its response shows.
  Why it matters: The baseline shows the specific instruction-following failure that fine-tuning is intended to improve.

Key terms: pretrained GPT model, gpt2-medium (355M), gpt2-small (124M), capacity, generate function, generated_text, input instruction, strip(), validation set, expected response

7.6 Fine-tuning the LLM on instruction data

This section fine-tunes a pretrained LLM on instruction data, interprets its training progress, and discusses practical limits of the run.

• Instruction fine-tuning setup: Describe how the section trains the loaded pretrained model on the prepared instruction dataset, reusing earlier loss and training functions and configuring the optimizer, epochs, evaluation frequency, and validation instruction.
  Why it matters: It connects the prepared instruction data and pretrained model to the actual fine-tuning run.
• Training and validation loss trends: Explain what the falling training and validation losses over two epochs suggest, including why the slower decline in the second epoch and the risk of overfitting matter.
  Why it matters: The loss curves help interpret whether fine-tuning is progressing and whether longer training is warranted.
• Generated response inspection: Explain how generated responses provide a check on the model's progress beyond loss values, using the validation sentence converted to passive voice as the section's example.
  Why it matters: A lower loss is encouraging, but the answer itself shows whether the model followed the instruction in the example.
• Fine-tuning hardware trade-offs: Describe the section's practical options when fine-tuning is too slow or runs out of memory: use a smaller model or GPU, and for the larger Alpaca exercise reduce batch size or allowed sequence length.
  Why it matters: These adjustments make the exercise more feasible on limited hardware.

Key terms: pretrained model, instruction dataset, data loaders, optimizer, epochs, evaluation frequency, training loss, validation loss, overfitting, generated responses

7.7 Extracting and saving responses

The section extracts and saves responses from a fine-tuned LLM’s held-out test set, then explains how those responses can be assessed and why they are saved for later evaluation.

• Test response extraction and comparison: Describe how the section generates responses for held-out test instructions, extracts each response from the combined input-and-output text, and compares initial examples with the expected answers.
  Why it matters: Extracting the answer itself makes the model’s output easier to inspect against the expected response.
• Instruction response evaluation choices: Distinguish instruction-response evaluation from classification accuracy and explain why human or automated conversational assessment is relevant to this test set.
  Why it matters: Choosing an evaluation approach depends on whether the aim is to test factual answers or the quality of conversation.
• Saved responses and model artifacts: Explain how the section prepares the full test set for later assessment and preserves the fine-tuned model for reuse.
  Why it matters: Saving both the answers and the model makes later evaluation and reuse possible without repeating this preparation.

Key terms: generate function, generated_text, test set, model response, classification accuracy, conversational performance, human evaluation, automated conversational benchmarks, test_set, model_response

7.8 Evaluating the fine-tuned LLM

The section replaces inspection of a few responses with an automated, reference-based evaluation of the instruction-fine-tuned LLM’s test-set responses.

• Reference Based Response Scoring: Explain how a larger instruction-fine-tuned LLM evaluates the model’s generated test-set responses against reference responses, and how integer scores become an average performance score.
  Why it matters: Averaged ratings make it practical to assess more than a handful of generated answers.
• Local Ollama Evaluation Setup: Describe Ollama’s role in running the Llama 3 evaluator locally and what must remain running for Python to query it.
  Why it matters: The automated evaluator needs a reachable local model service, not another training run.
• Score Interpretation And Variability: Explain what the reported average score is useful for and why repeated evaluations can give a more robust result.
  Why it matters: Treating the score as a comparative benchmark rather than an exact constant makes experiments easier to interpret.
• Performance Improvement Options: Describe the kinds of changes the section suggests trying to improve the fine-tuned model’s performance.
  Why it matters: The benchmark gives you a way to compare these possible improvements.

Key terms: Llama 3, test set response, reference, integer scores, average score, Ollama, ollama serve, ollama run, REST API, query_model

7.9 Conclusions

This conclusion recaps the three main stages of developing an LLM before turning to what to explore next.

• LLM development cycle stages: State the main stages of the LLM development cycle covered in the book: implementing an LLM architecture, pretraining the model, and fine-tuning it for specific tasks.
  Why it matters: These stages give you a concise way to recap how the book builds an LLM.

Key terms: LLM development cycle, LLM architecture, pretraining, fine-tuning

7.9.1 What’s next?

This subsection identifies preference fine-tuning as an optional step after instruction fine-tuning and points readers to supplementary resources.

• Optional preference fine-tuning: Explain when preference fine-tuning can be used and what it is useful for.
  Why it matters: It distinguishes an optional customization step from the instruction fine-tuning already covered.

Key terms: instruction fine-tuning, preference fine-tuning, user preferences

7.9.2 Staying up to date in a fast-moving field

The section suggests ways to stay informed as AI and LLM research changes rapidly.

• Sources for LLM updates: Describe how to keep up with rapidly evolving AI and LLM research by reading recent papers and following researchers and practitioners who discuss new developments online.
  Why it matters: Keeping up with current papers and community discussions helps you notice developments after the book was written.

Key terms: arXiv, X, Reddit, r/LocalLLaMA

7.9.3 Final words

The author closes by contrasting the educational value of building an LLM from scratch with tools readers might explore for real-world applications.

• Educational and practical LLM paths: Explain why the author favors building an LLM from scratch for learning, and what the author recommends exploring for real-world applications.
  Why it matters: It clarifies the difference between the book’s hands-on learning approach and the author’s suggestion for exploring practical tools.

Key terms: LLM from scratch, pretraining, fine-tuning, Axolotl, LitGPT

7.10 Summary

Summarizes how a pretrained LLM is prepared, fine-tuned, and evaluated for following instructions.