Chapter 5: Training large language models: How to generate the generator

Introduction

The chapter introduces training an LLM on a prepared dataset and previews the environment, training methods, and practical advice it will cover.

• Slack Dataset Persona: Describe how the chapter will use the Slack-message dataset prepared in the previous chapter to create an LLM persona resembling the reader.
  Why it matters: This gives the training work a concrete purpose.
• Training Chapter Path: Explain why the chapter begins with a suitable training environment, then moves from training from scratch and finetuning to methods for making training more efficient, faster, and cheaper, and ends with field-tested tips.
  Why it matters: The sequence shows why resources and setup come before hands-on training techniques.

Key terms: training dataset, Slack messages, persona, training environment, resource-demanding, training from scratch, finetuning, advanced methodologies, tips and tricks

5.1 Multi-GPU environments

This section explains why training and working with large LLMs calls for multi-GPU environments and introduces cloud access and supporting tools as the next steps.

• Reasons for multiple GPUs: Explain why an LLM that runs inference on one GPU may need many GPUs for training, and why the largest models require more than one GPU even for inference.
  Why it matters: The hardware needed to run a model can be very different from what is needed to train it or work with the largest models.
• Cloud environment starting point: Describe the section’s proposed starting point: get access to a cloud multi-GPU environment, then use tools and libraries to work with it as a playground for the chapter.
  Why it matters: The later training techniques need an environment in which readers can practice with large models.

Key terms: multi-GPU environment, inference, training, parallelize, cloud, tools and libraries, playground

5.1.1 Setting up

This section walks through setting up, accessing, and deleting a paid multi-GPU Google Cloud VM for LLM work.

• GPU costs and prerequisites: Explain why the book's small examples do not require a multi-GPU VM, when a larger rig is useful, and what must be in place before creating one on Google Cloud.
  Why it matters: A multi-GPU VM can speed up larger experiments, but it incurs charges and requires GPU access to be enabled.
• VM configuration and creation: Describe how the section chooses a GPU machine type, region and zone, and prepared deep-learning image, then creates a VM with appropriate disk space, authorization, and NVIDIA drivers.
  Why it matters: These choices determine whether the VM is available, large enough for LLM assets, and ready to use its GPUs.
• VM access and cleanup: Trace the workflow for checking that a created VM is running, connecting by SSH, copying project files with SCP, and deleting the instance when finished.
  Why it matters: Creating the VM is only useful if you can get code onto it, and deleting it prevents unnecessary charges.
• VS Code remote connection: Describe how to use VS Code's Remote-SSH extension and the gcloud SSH configuration to work on the VM through an IDE instead of only a terminal.
  Why it matters: Remote-SSH lets you use a familiar editor while working on the cloud machine.

Key terms: multi-GPU, Google Cloud Project, billing, gcloud CLI, gpus_all_regions quota, g2-standard-24, NVIDIA L4 GPU, region, zone, deep-learning image

5.1.2 Libraries

The section compares four libraries for multi-GPU LLM work and explains why compatible software environments matter as much as correctly configured hardware.

• Software compatibility and performance: Explain why a multi-GPU setup needs compatible CUDA, PyTorch, and library versions, not just working hardware. Describe how a mismatch can make training drastically slower and why the environment is worth tracking carefully.
  Why it matters: A software mismatch can erase the performance gains expected from expensive hardware.
• Distributed training library roles: Distinguish DeepSpeed’s distributed deep-learning optimizations and ability to scale across many GPUs from Accelerate’s simpler way to adapt a PyTorch training loop for parallel, multi-GPU use.
  Why it matters: The libraries overlap, but their different approaches affect how you build a training workflow.
• Quantization and attention libraries: Describe BitsandBytes as a way to use lower-bit quantization and efficient matrix operations, including 8-bit components, and xFormers as a source of PyTorch-like components such as memory-efficient exact attention. Explain that these optimizations target speed or memory use while retaining useful model fidelity.
  Why it matters: These tools address different computational bottlenecks without treating speed as the only consideration.

Key terms: CUDA Toolkit, PyTorch, multi-GPU instances, DeepSpeed, Accelerate, distributed deep learning, parallelizing, BitsandBytes, xFormers, quantization

5.2 Basic training techniques

This section introduces the choices, resource needs, and staged life cycle behind basic LLM training techniques.

• Initial training choices: Describe how LLM training begins by choosing an architecture, data, and objective: typically a Transformer, a large and diverse text corpus, and next-word prediction.
  Why it matters: These choices shape what the foundation model can learn before it is specialized.
• Training resource differences: Distinguish the VRAM demands of training from scratch from those of finetuning, including the section's rough sizing examples and why finetuning can use less.
  Why it matters: The resource difference affects which training approach is practical.
• Staged LLM training life cycle: Trace the progression from training a foundation model on a large, often unrefined corpus, through finetuning it on a smaller curated dataset, to prompting it for a desired response. Explain that prompting is not traditional training and that improved models can help improve later training datasets.
  Why it matters: The life cycle connects the three basic techniques and shows how model and dataset improvements can feed into each other.

Key terms: Transformer architecture, diverse and representative text data, predict the next word, VRAM, batch sizes, context length, quantization, foundation model, transfer learning, finetuning

5.2.1 From scratch

This section walks through building and training a language model from scratch, then uses a small character-based example to show both the training loop and the limits of its tokenizer.

• From-scratch model preparation: Describe what starting from scratch entails: defining an architecture rather than reusing one, assembling and tokenizing a diverse dataset, and initializing new weights. Explain how the task, data size, and compute resources guide the architecture choice.
  Why it matters: These choices determine what the model can learn and whether training it is practical.
• Next-token training loop: Trace how the model predicts the next token from preceding tokens, compares its prediction with the training text to calculate loss, and adjusts its weights through backpropagation and an optimizer as it processes batches over epochs.
  Why it matters: This loop is how a newly initialized generator learns from text.
• Validation and generalization: Explain why minimizing training loss alone is insufficient and how monitoring performance on separate validation data helps detect overfitting. Describe the section's suggested responses, such as early stopping, dropout, learning-rate scheduling, or revisiting model parameters and architecture, without treating them as guarantees.
  Why it matters: The aim is useful text generation in new contexts, not just good predictions on the training data.
• Character tokenizer tradeoff: Explain why the example uses character-based tokenization for simplicity yet finds character-level modeling weak for language modeling, and identify broader token units and pretraining as ways to capture more information.
  Why it matters: The example shows how a convenient tokenization choice can limit what a model captures from text.

Key terms: from scratch, architecture, Transformer, dataset, tokenized, randomly initialized, autoregressive, batches, epochs, loss

5.2.2 Transfer learning (finetuning)

This section explains how finetuning adapts a pretrained LLM to a specific task, then illustrates the approach with GPT-2 and OpenAI’s finetuning API.

• Pretraining and task adaptation: Explain how transfer learning reuses language patterns learned from a large, general-purpose corpus by updating a pretrained model on a smaller, task-specific dataset. Contrast its computational demands and likely task performance with training a model from scratch.
  Why it matters: This distinction explains why finetuning is practical for many LLM applications.
• Finetuning preparation and evaluation: Describe the section’s finetuning workflow: choose a pretrained model for the task and available resources, prepare compatible task-specific data, adjust the model on that data, and evaluate it on separate unseen examples. Explain why a smaller learning rate is typically used.
  Why it matters: The workflow connects model selection and data preparation to a check that the adapted model generalizes.
• GPT-2 example and limits: Explain what the section’s GPT-2 comparison illustrates: finetuning reuses an existing architecture, tokenizer, weights, and embeddings rather than defining them anew, and the finetuned model improves on the from-scratch example. State the section’s caution about longer narratives.
  Why it matters: The example shows both the practical shortcut provided by pretraining and a remaining limitation.
• OpenAI finetuning job flow: Trace the section’s OpenAI API workflow: upload a correctly formatted training file for finetuning, create a job specifying that file and a model, retrieve the resulting model name, and use it for completions. Describe the tradeoff between the convenience of this hosted process and giving OpenAI control over how finetuning occurs.
  Why it matters: The hosted example offers a simple route to finetuning while clarifying what control the user gives up.

Key terms: transfer learning, source domain, target domain, pretraining, finetuning, pretrained model, preprocessed, tokenized, learning rate, separate dataset

5.2.3 Prompting

This section introduces prompting as a way to guide an LLM at run time and explains why crafting the prompt matters.

• Run-time guidance through prompts: Explain that a prompt is the initial input giving an LLM context or instructions for a task, such as translation or summarization, and that the model responds to different tasks based on that input.
  Why it matters: Prompting lets one model handle different requests by changing what you ask it to do.
• Effective prompt crafting: Describe prompt engineering as crafting prompts to produce more useful outputs, including being explicit, providing an example of the desired output, or trying different phrasing. Explain that small wording changes can substantially change the response.
  Why it matters: The way a request is phrased can make a substantial difference to the answer you get.

Key terms: prompt, prompting, run time, prompt engineering, model’s capabilities and limitations

5.3 Advanced training techniques

Introduces why production LLMs may need different advanced training techniques and reviews four classical learning paradigms used to understand them.

• Reasons for advanced techniques: Explain that advanced training techniques address different production needs, including better generated text, smaller models, continuous learning, faster training, and lower costs; the choice depends on the organization’s needs.
  Why it matters: The reason for changing a training approach depends on the production problem you are trying to solve.
• Four classical learning paradigms: Distinguish supervised learning by its labeled expected outputs, unsupervised learning by finding similarities without labels, adversarial learning by a forger and critic competing, and reinforcement learning by rewards based on a model’s actions rather than predefined labels.
  Why it matters: These paradigms provide the background for understanding the advanced training approaches that follow.

Key terms: advanced training techniques, production-ready model, supervised learning, unsupervised learning, adversarial learning, Critic model, Forger model, reinforcement learning

5.3.1 Prompt tuning

This section explains how prompt tuning supplies task context during training to improve performance while reducing the need to finetune a large language model.

• Task context during training: Explain why giving a language model the expected task and relevant information during training can work better than making it infer the task from input–output examples alone.
  Why it matters: Task context helps steer a model toward the output you want.
• Efficient model adaptation: Describe the section’s claimed benefits of prompt tuning: needing less data to converge and adapting a frozen language model to new tasks without full retraining or finetuning.
  Why it matters: Avoiding extensive changes to a large model can make task adaptation more practical.
• BLOOMZ prompt tuning example: Describe how the BLOOMZ example differs from the earlier finetuning example: it prepends a task instruction to each input and uses Parameter-Efficient Fine-Tuning to reduce memory requirements.
  Why it matters: The example connects the prompt-tuning idea to a concrete training setup.

Key terms: prompt tuning, pragmatic context, expected task, frozen language model, converge, finetuning, BLOOMZ, Parameter-Efficient Fine-Tuning, PEFT, prepending a prompt

5.3.2 Finetuning with knowledge distillation

This section explains how knowledge distillation trains a smaller LLM to imitate a larger one, trading some performance for substantial savings in size and computation.

• Teacher student distillation process: Explain how a pretrained, larger teacher model guides a smaller student on the same task: the student learns from task labels and from the teacher’s output probability distributions, with a loss term encouraging their predictions to align.
  Why it matters: The teacher’s outputs are what distinguish distillation from ordinary finetuning.
• Distilled model efficiency tradeoff: Describe why distillation produces a smaller, faster model for resource- or response-time-limited settings, while acknowledging that performance may fall too—typically much less sharply than model size. State that the student can be finetuned further for a specific task or dataset.
  Why it matters: The value of distillation depends on the balance between efficiency gains and lost performance.
• Distillation and human feedback: Distinguish distillation’s learning from a teacher’s predictions on existing data from the subsequent RLHF approach, which uses human interactions and feedback to refine a model.
  Why it matters: The distinction clarifies why the section presents RLHF as a progression beyond distillation.

Key terms: knowledge distillation, teacher model, student model, soft targets, loss function, compact student model, computational cost, finetuned, RLHF, human feedback

5.3.3 Reinforcement learning with human feedback

This section explains how human preferences can guide reinforcement learning for LLMs, then weighs RLHF’s benefits against its costs and production risks.

• Reward design pitfalls: Explain why hand-designed rewards can produce behavior that satisfies the reward without accomplishing the intended task, using the StarCraft Pylon example.
  Why it matters: This failure explains why human feedback is appealing in the first place.
• Human preference training loop: Describe how RLHF uses human choices between attempted outputs to guide training instead of relying on a manually specified reward for the task. For LLMs, the human compares generated responses to a prompt; the figure depicts a reward model and proximal policy optimization (PPO).
  Why it matters: The comparison between responses is the central idea behind RLHF in this section.
• Preference versus factual correctness: Distinguish making answers acceptable to human reviewers from making them factually correct, and explain how reviewer expertise, bias, privacy, or unreliable user feedback can affect RLHF.
  Why it matters: A preferred answer is not necessarily a correct one.
• Continual update degradation: Explain why continual RLHF updates can degrade a production model, including how evaluation prompts can leak into feedback-based training and why a changing model can disrupt prompting or RAG workflows.
  Why it matters: Teams need to weigh easy ongoing updates against trustworthy evaluation and stable application behavior.
• RLHF application tradeoffs: Justify when the section considers RLHF useful—such as refining a chatbot’s tone or a summarizer’s output—and why it nevertheless advises against using it here, citing computational cost and limited improvement relative to high-quality supervised datasets.
  Why it matters: The section’s practical takeaway is about choosing when human preference is worth its costs and risks.

Key terms: reward system, intermediate rewards, Pylons, reinforcement learning with human feedback, RLHF, reward model, proximal policy optimization, PPO, human-acceptable answers, domain experts

5.3.4 Mixture of experts

This section explains how a mixture of experts uses sparse routing to retain specialization while reducing computation, then describes finetuning an existing Switch Transformer model.

• Sparse expert routing: Explain that a mixture of experts starts with multiple experts that specialize during training, then routes a tokenized input to relevant experts rather than necessarily activating them all.
  Why it matters: Selective activation is the central reason an MoE can combine specialization with lower computational demands.
• Specialization and computation tradeoff: Justify why consulting experts whose learned specialization matches the input can substantially reduce inference computation while retaining specialized knowledge, provided training data is sufficiently representative.
  Why it matters: This tradeoff explains why the section presents MoE as useful for large models.
• Switch Transformer finetuning: Describe the section’s approach of starting with an existing Switch Transformer MoE and finetuning it on a dataset using Hugging Face’s API; explain that MoE finetuning typically updates the gating mechanism and experts while keeping shared parameters intact, with the configured trainer handling that complexity.
  Why it matters: It distinguishes using an existing MoE from building one and identifies what makes its finetuning different.
• Switch Transformer design aims: State that the Switch Transformer addressed model size and training instability by simplifying expert routing and enabling training at lower precision, here bfloat16.
  Why it matters: These design choices help explain why the existing model makes the example practical.
• Parameter efficient finetuning preview: Distinguish the section’s brief preview of PEFT, which aims to reduce finetuning size and computational demands, from LoRA, which uses low-rank factorization to reduce the number of parameters while maintaining or improving performance.
  Why it matters: It connects MoE finetuning to the resource-efficient approaches introduced next.

Key terms: mixture of experts, MoE, sparsity, experts, routing algorithm, representative-enough dataset, tokenized input, computational requirements, specialization, Switch Transformer

5.3.5 LoRA and PEFT

This section explains how LoRA uses small, low-rank adapters to adapt a foundation model for new tasks or domains without storing a separately finetuned copy of the whole model.

• Low-rank weight updates: Explain how LoRA represents a weight update ΔW as the product of two smaller matrices, Wa and Wb, rather than training and storing a full-size update.
  Why it matters: This factorization is what makes a LoRA adapter small while still changing the model’s output.
• Adapter rank tradeoff: Explain that LoRA’s rank R is the shared inner dimension of its two matrices, and describe the tradeoff between a higher rank’s capacity to match the original model’s accuracy and its lower memory savings.
  Why it matters: Rank is a practical choice that balances task complexity against memory use.
• Base model and domain adapters: Describe how a LoRA runs alongside a retained foundation model and why separate small adapters make adaptation to multiple tasks or domains more feasible than storing multiple fully finetuned models.
  Why it matters: Keeping one base model and separate adapters can reduce storage while keeping domain-specific changes apart.
• Illustrative LoRA configuration: Describe the section’s example as a LoRA configured at rank 16 for RoBERTa-Large token classification on the bionlp dataset, with the adapter used alongside the base model for inference.
  Why it matters: The example connects the rank and base-model concepts to a concrete adaptation.

Key terms: LoRA, ΔW, Wa, Wb, rank, R, c-value, LoraConfig, foundation model, base model

5.4 Training tips and tricks

This introduction explains why the section offers practical, hard-won tips for training and finetuning LLMs.

• Purpose of training tips: Explain that finetuning on your own data can help teach an LLM appropriate guardrails, and that the section offers practical tips for both training and finetuning rather than focusing on research into new models.
  Why it matters: It frames the upcoming advice as help with parts of training that practitioners often learn the hard way.

Key terms: finetuning, guardrails, training

5.4.1 Training data size notes

This section gives rough data-size checks for training a foundation model or finetuning an existing LLM, and warns against expecting strong results from too little data.

• Foundation model data scale: Explain the section’s rule of thumb that training a foundation model calls for roughly 20 tokens per parameter, and that having fewer tokens raises the risk of overfitting.
  Why it matters: The amount of available data can determine whether training a model from scratch is a sensible plan.
• Finetuning data minimum and alternatives: Describe the section’s much smaller, experience-based data minimum for finetuning an existing model—illustrated as about 10K tokens for a 1B-parameter model—and explain why very small datasets may not justify finetuning. State the suggested alternatives: LoRA, RAG, or both.
  Why it matters: Checking whether there is enough data can prevent spending on finetuning that has little effect.

Key terms: foundation model, parameters, tokens, overfitting, finetune, 10K tokens, LoRA, RAG

5.4.2 Efficient training

This section explains how training-loop choices trade speed against memory use and why those choices must fit production constraints.

• Speed and memory tradeoff: Explain that training choices generally cannot maximize both speed and memory efficiency: using less memory can let a larger model fit, but often makes training slower. Describe gradient accumulation and checkpointing as examples that can substantially reduce memory use at the cost of training time.
  Why it matters: The right training setup depends on whether speed or fitting a larger model matters more.
• Batch size and deployment constraints: Explain why the section recommends a power-of-two batch size for speed and memory efficiency, while checking whether production imposes a different batch size. Describe how the example's fixed production batch size of 20 undermined an optimization achieved in development.
  Why it matters: An optimization in development may not deliver its intended benefit if production runs under different constraints.

Key terms: speed, memory utilization, gradient accumulating, checkpointing, batch size, power of 2, production environment, training loop

5.4.3 Local minima traps

This section explains how to recognize suspicious early convergence during LLM training and use checkpoints and learning-rate changes to get past a local minimum.

• Suspicious early convergence: Explain why apparent early convergence may be a local-minimum trap rather than a finished model, and why it calls for careful testing—especially if the model has not trained on the full dataset.
  Why it matters: A model that appears to have converged can still have substantial training left to do.
• Checkpoint and learning-rate recovery: Describe how to retry a run that repeatedly converges early: return to a checkpoint from roughly 100 steps before the problem, lower the learning rate until the model gets past it, then raise the rate again and continue while retaining checkpoints for recovery.
  Why it matters: This gives you a way to continue training past a recurring trap without losing your recovery points.

Key terms: local minima, converging early, full dataset, checkpoint, learning rate, local minimum

5.4.4 Hyperparameter tuning tips

This section advises prioritizing better data and model optimizations over hyperparameter tuning, while noting when tuning may still be useful.

• Data-first metric improvement: Explain why getting more data, cleaning it further, or better representing a target metric in the dataset is usually more useful than hyperparameter tuning for improving an LLM’s results. Describe oversampling as one possible way to improve that representation.
  Why it matters: It helps you choose the intervention most likely to improve the result you care about.
• Limited role of hyperparameter tuning: Describe hyperparameter tuning as an option for a small accuracy or F1 gain, particularly when the dataset cannot be changed soon, rather than a main route to better LLM performance. State that the section suggests Optuna for tuning and contrasts tuning with quantization and other optimizations.
  Why it matters: It helps you decide when a modest tuning effort is worthwhile and when to pursue a larger improvement elsewhere.

Key terms: hyperparameter tuning, accuracy, F1 score, oversampling, Optuna, quantizing the weights, F1

5.4.5 A note on operating systems

This section recommends learning Linux for professional LLM work despite its difficult setup, and identifies the tools that make that work practical.

• Operating system tradeoffs: Explain why the section favors Linux over Windows and macOS for professional LLM work, while acknowledging that Linux can be frustrating to set up.
  Why it matters: Your operating system affects how easily you can use the hardware and software needed for LLM work.
• Linux setup skills: Describe the skills and tools the section suggests learning to work effectively with LLMs on Linux, including the command line, GPU setup, and supporting environments.
  Why it matters: These skills help you get past Linux’s initial setup difficulties.

Key terms: Windows Subsystem for Linux, macOS, Linux, Bash, command line, CUDA Toolkit, Nvidia drivers, virtual environments

5.4.6 Activation function advice

The section advises against casually changing a pretrained LLM’s activation function and points from model training toward production serving.

• Activation function change tradeoffs: Explain why changing an LLM’s activation function is usually unnecessary outside model-performance research, and why changing one in a pretrained model means repeating pretraining.
  Why it matters: An activation-function swap is not a simple way to improve an already pretrained model.
• From training to serving: Describe the chapter’s handoff from setting up and training or finetuning an LLM to making a model production-ready and serving online inference.
  Why it matters: Creating a model is only part of the work needed to make it usable as an online service.

Key terms: activation functions, GPT-2, GELU, GeGLU, pretraining, finetuning, production-ready, LLM service, online inference

5.5 Summary

The chapter summary recaps the training workflow, the roles of data and adaptation techniques, and the resource tradeoffs involved in training LLMs.