Chapter 9: Creating an LLM project: Reimplementing Llama 3

Introduction

The chapter introduces a hands-on project that brings earlier LLM lessons together by implementing and training a Llama-inspired model, improving it, and deploying it.

• End-to-end project scope: Describe the project as a from-scratch, hands-on exercise that brings together earlier lessons: implementing Meta’s Llama 3, pretraining a simple model, improving it, and serving it. Explain that it roughly follows a research paper but takes shortcuts rather than examining the research in depth.
  Why it matters: This scope explains why the chapter emphasizes completing a working project over reproducing research exactly.
• Production preparation and deployment: Trace the planned path from training to a shareable production endpoint: use quantization to prepare the model for serving, finetune it for a specific task with low-rank adaptation (LoRA), and deploy it.
  Why it matters: The project is meant to show not just model training, but the steps toward a usable, shareable deployment.

Key terms: Meta’s Llama3, pretraining, from scratch, quantization, low-rank adaptation (LoRA), production endpoint

9.1 Implementing Meta’s Llama

This section explains why Llama 3 is a useful model to study and sets expectations for building and productionizing a smaller reimplementation.

• Llama 3 availability and data: Explain why the section highlights Llama 3 despite its not usually outperforming much larger models: its availability supports a community that builds tooling, optimizations, and data, while its clean training data illustrates the importance of data over architecture.
  Why it matters: This explains why Llama 3 is worth implementing even though it is not the largest or strongest model.
• Reimplementation scope and purpose: Describe the chapter’s plan to build a real but below-average Llama-style model and productionize it, rather than train a Meta-quality model from scratch. Explain that limited data and GPUs constrain its quality, while the exercise exposes implementation steps and likely problems.
  Why it matters: The modest model is a way to learn the full project workflow rather than merely obtain impressive weights.

Key terms: Llama 3, availability, open source software community, clean data, real model, data, GPUs, productionize

9.1.1 Tokenization and configuration

This section sets up the tokenizer and master configuration for a from-scratch Llama project and explains how those choices affect training and inference.

• Tokenizer choice and model capability: Explain why the project starts with Llama’s tokenizer even though the model has no pretrained weights, and how tokenization choices affect what the model can see during training and inference. Describe when a dataset-specific tokenizer or added domain tokens might be preferable.
  Why it matters: A tokenizer that fits the data and task gives the model a better starting point.
• Training tokenizer preparation: Describe why the Hugging Face inference tokenizer needs a padding token for this project’s training setup, and identify the vocabulary and encoding and decoding helpers prepared for later batch processing.
  Why it matters: These preparations let the training pipeline handle the tokenizer’s outputs in batches.
• Master configuration tradeoffs: Explain that the master config controls model parameter count, training duration, and memory use per dataset row. Describe why its small defaults accommodate modest hardware, including CPU-only training, while leaving room to experiment.
  Why it matters: The config lets readers scale the exercise to the hardware they have.

Key terms: Llama 3 tokenizer, SentencePiece, tiktoken, pretrained weights, padding token, <PAD>, vocab, encoding, decoding, batch processing

9.1.2 Dataset, data loading, evaluation, and generation

This section sets up memory-conscious data preparation, next-token training batches, validation loss, and text generation for a small Llama project.

• Dataset preparation and streaming: Explain why dataset quality and distinctive data sources matter, and describe how the TinyStories dataset is streamed, minimally cleaned, tokenized, and shuffled for training without leaking data between splits.
  Why it matters: The model depends on the quality of its data, while the loading method determines whether preparation fits in available memory.
• Shifted next-token training batches: Describe how get_batches makes self-supervised input and expected-output sequences by taking a context window of tokens and shifting the expected output one token to the right, so the model learns to predict the next token without manual labels.
  Why it matters: The input–output shift defines the text-completion task the model trains on.
• Validation loss and its limits: Explain that evaluation averages loss over some batches to produce validation loss, while recognizing that this useful summary does not fully represent model performance or necessarily reflect what matters to users.
  Why it matters: A convenient training signal should not be mistaken for a complete assessment of the model.
• Autoregressive generation and sampling: Trace how a tokenized prompt is repeatedly passed through the model, the last-position logits are temperature-scaled, and a next token is selected and decoded. Distinguish top-k or top-p candidate filtering from the choice between argmax and multinomial sampling for selecting a token.
  Why it matters: These choices turn model outputs into text and affect how exploratory the generated text can be.

Key terms: TinyStories, streaming, tokenization, attention masks, train and val splits, data leaks, get_batches, self-supervised learning, context window, expected output

9.1.3 Network architecture

The section builds and trains a small feed-forward language model as a baseline, explains its loss and padding choices, and uses its results to set up the move toward Llama.

• Feed-forward baseline architecture: Describe the baseline as two linear layers with a ReLU activation between them, producing tokens before the chapter introduces changes toward Llama. Explain that it approximates rather than fully replicates Llama 3.
  Why it matters: This simple model gives you a point of comparison for each later architectural change.
• Cross-entropy padding exclusion: Explain why the model uses cross-entropy loss and why it sets ignore_index to the tokenizer's pad_token_id. Describe how frequent padding tokens could otherwise reward the model for predicting padding instead of useful next tokens.
  Why it matters: Ignoring padding keeps the training signal focused on actual text rather than an artifact of batching.
• Baseline training and results: Describe how the training loop uses the master configuration's epochs setting for passes that are more like steps, and how d_model can change the model's parameter count. State what the short run shows: a smooth loss curve and generated tokens that nearly make sense, while more full-dataset epochs are encouraged.
  Why it matters: The short run demonstrates a working language model without treating its early results as the limit of what it could do.

Key terms: two-layer feed-forward network, linear model, ReLU, Llama 3, cross entropy, perplexity, <PAD>, ignore_index, pad_token_id, training loop

9.2 Simple Llama

The section builds a resource-conscious Llama-like model, explains how it differs from the earlier feed-forward network, and assesses what its training results show.

• Simplified Llama design scope: Explain that Simple Llama resembles Llama 3 but does not reproduce Meta’s released weights and layers exactly. Describe the two reasons for simplifying it: making the implementation approachable and keeping it runnable in Kaggle or Colab.
  Why it matters: The model is an exercise in understanding Llama-like design, not an exact reproduction of Meta’s release.
• Llama block architecture: Describe how Simple Llama replaces the earlier feed-forward arrangement with multiple LlamaBlocks set by `n_layers`, using RMSNormalization, RoPEMaskedMultiheadAttention, and SwiGLU rather than ReLU. Trace the forward path in broad terms through normalization, attention, another normalization, and the output logits.
  Why it matters: These changes are what make the exercise a Llama-like network rather than just the earlier feed-forward model.
• Training learning rate scheduler: Explain that this version adds a scheduler to adjust the learning rate during training according to a schedule, helping address potential exploding gradients and speed convergence.
  Why it matters: The scheduler is a training change alongside the architectural changes.
• Scaling and training outcomes: Describe the reported result of scaling Simple Llama to over 180 million parameters: generated text guesses punctuation more often but usually places it incorrectly, and loss is higher. Explain why the section does not treat this as the final verdict: training has not sufficiently covered the dataset, and more data, context, layers, compute, and training time could improve the model.
  Why it matters: The results separate having built a Llama-like model from having trained a good text generator.

Key terms: Simple Llama, Llama 3, Kaggle, Colab, LlamaBlocks, n_layers, RMSNormalization, RoPEMaskedMultiheadAttention, SwiGLU, ReLU

9.3 Making it better

This section uses a model that passes internal tests but fails production checks to shift attention toward deployability, model size, and the systems surrounding the model.

• Production Readiness Beyond Benchmarks: Explain why passing internal efficacy tests does not mean a model is ready for production: the tests may not represent production conditions, and the model may be too large or slow even to complete production tests. Describe model size as a concern after efficacy testing.
  Why it matters: A model can perform well in internal tests yet be impractical to deploy.
• Systems Around The Model: Explain that the model alone is not the whole source of production success: the engineered systems around it, including data, can make the difference. Distinguish this section’s deployment-focused advice from teaching how to pretrain a working language model.
  Why it matters: Improving the model alone may not solve the problems that determine whether it succeeds in production.

Key terms: internal benchmarks, production environment, efficacy tests, size, systems we engineer around models, data, pretraining, deploy

9.3.1 Quantization

Shows how post-training INT8 quantization can shrink the Llama model for deployment while warning that compression can make its output less reliable.

• Post-training INT8 model compression: Explain why the model’s disk size is a deployment problem and describe using PyTorch to quantize the trained model to INT8. State that the example compares disk sizes before and after quantization.
  Why it matters: A smaller model can fit production environments that cannot serve the original model.
• Compression and output reliability tradeoff: Explain that compressing weights further can help a model fit a chosen production environment, but increases perplexity and can make LLM behavior less stable and predictable, even with good prompt engineering.
  Why it matters: A model that fits the deployment environment still needs to produce reliable results.

Key terms: quantization, INT8, PyTorch, compression, perplexity, compress weights, prompt engineering

9.3.2 LoRA

This section uses LoRA to adapt Simple Llama for a new task without retraining it from scratch, then explains how to save and use the adaptation for inference.

• LoRA adaptation rationale: Explain why LoRA is a useful response when data drift reveals a task the model handles poorly, but the team lacks the time, money, or new data for full retraining or effective finetuning.
  Why it matters: The reason for choosing LoRA determines when this approach is worth trying.
• LoRA training outputs: Describe the section's LoRA training setup: add LoRA to Simple Llama, train with a new instruct dataset and get_batches function, and save separate state dicts for the model and LoRA. State that the LoRA file is typically small, with its size depending on rank.
  Why it matters: Keeping the adaptation separate makes task-specific changes practical to save and reuse.
• LoRA inference choices: Distinguish loading a saved LoRA on top of the original model from merging its layers into the model for inference, and identify merging as the section's choice. Describe the observed change in generated text and how removing LoRA restores the original functionality.
  Why it matters: The inference choice affects how the adaptation is used and whether the original behavior can be recovered.

Key terms: LoRA, data drift, finetune, LoRALayer, SimpleLlama, instruct dataset, get_batches, state dicts, rank, state dict

9.3.3 Fully sharded data parallel–quantized LoRA

This section combines QLoRA with PyTorch’s fully sharded data parallel training to fine-tune large models across GPUs, then explains how to prepare the resulting model for upload.

• Quantized LoRA memory savings: Explain that QLoRA quantizes a model, freezes those quantized weights, and trains LoRA on top, reducing the memory needed to fine-tune a large model.
  Why it matters: QLoRA makes fine-tuning models that exceed a single GPU’s memory more practical and less expensive.
• FSDP distributed training role: Describe how fully sharded data parallel training adds data and model-parameter parallelism across multiple GPUs, handling sharding and rejoining so QLoRA can use consumer hardware.
  Why it matters: FSDP lets the training job draw on several GPUs rather than requiring one enterprise GPU with enough memory.
• Training output and upload preparation: Describe the section’s runnable FSDP-QLoRA approach and its output: a safetensors state dict that must be converted into a full model or checkpoint before upload for loading with classes such as AutoModel or LlamaForCausalLM.
  Why it matters: A successful training run does not by itself produce a model file that those loading classes can use.

Key terms: QLoRA, quantized model, frozen, LoRA, fully sharded data parallel, FSDP, data parallelism, model parameter parallelism, sharding, fsdp_qlora

9.4 Deploy to a Hugging Face Hub Space

This section shows how to prepare a trained Llama model for Hugging Face, set up a Space to host an app, and account for the compute its model needs.

• Spaces as hosted applications: Explain that Hugging Face Spaces are hosted containers where people can try a model through an app, with free and paid compute tiers and support for frontends such as Gradio, Streamlit, and FastAPI.
  Why it matters: A Space gives users a way to interact with a hosted model and can also serve as a production deployment option.
• Model conversion and Hub publishing: Describe converting the trained model's weights and directories into a format that can be pushed to the Hugging Face Hub, and explain that a logged-in user can call model.push_to_hub() to publish the model repository.
  Why it matters: Preparing and publishing the model makes it available for the intended Hugging Face deployment or sharing workflow.
• Space setup and code maintenance: Describe choosing compute based on the app's memory and GPU needs, then choosing whether to maintain the Space's code with Git or huggingface-cli. Explain that the Git route involves creating the Space, cloning it, and adding, committing, and pushing changes.
  Why it matters: A Space needs both suitable compute and a way to keep its application code updated.
• GPU requirements and hosting options: Explain that the chapter's models, especially the quantized ones, require GPUs to load and run and therefore errored on the free Space tier in the authors' attempt. Describe upgrading to a paid tier or using ZeroGPU, which provisions a GPU only when a Gradio app requests it for a task.
  Why it matters: The model's actual hardware requirements determine whether a proposed Space deployment will run.

Key terms: Hugging Face Spaces, Gradio, Streamlit, FastAPI, weights, Hugging Face Hub, model.push_to_hub(), simple Llama LoRA, Space, Git

9.5 Summary

The section recaps the model-design, adaptation, and deployment choices involved in building a Llama project.