liveBook · Manning

What you will learn in Rearchitecting LLMs

Structural techniques for efficient models

by Pere Martra · Manning Early Access Program (MEAP)

Chapter by chapter

Chapter 1: Why rearchitecting LLMs matters

The introduction explains why broad, generic LLMs can be a poor fit for specialized tasks and presents structural rearchitecting as an alternative to relying only on prompts.

1.1 Current challenges to scaling LLMs · 1.1.1 Operational costs · 1.1.2 Competitive differentiation · 1.1.3 Explainability · 1.2 The solution: the model rearchitecting pipeline · 1.2.1 Structural optimization · 1.2.2 Knowledge distillation · 1.2.3 Specialization · 1.2.4 Optional: Teacher correction · 1.2.5 How data moves through the pipeline · 1.3 Toolkit and techniques · 1.4 Your LLM rearchitecture roadmap · 1.5 Summary

What you learn in chapter 1 →

Chapter 2: An end-to-end rearchitecting project

Introduces a general-purpose project to make a base model smaller and faster while preserving its capabilities through structural optimization and knowledge recovery.

2.1 The rearchitecting workflow · 2.2 Establishing the baseline · 2.3 Applying depth pruning · 2.4 Evaluating the impact of pruning · 2.5 Recovering knowledge · 2.6 Analyzing the final result · 2.7 Hands-on lab · 2.8 Summary

What you learn in chapter 2 →

Chapter 3: A blueprint to modern transformers

The introduction identifies attention memory growth and MLP compute as distinct bottlenecks that motivate studying transformer components.

3.1 Classical architecture: DistilGPT2 · 3.1.1 General behaviour of a Transformer model · 3.1.2 Classical attention mechanism · 3.1.3 The classic MLP mechanism · 3.1.4 The Transformer dimensions: depth and width · 3.2 The modern Transformer architecture · 3.2.1 Optimized attention: from Multi-Head Attention (MHA) to Grouped-Query Attention (GQA) · 3.2.2 The evolution of the MLP: from simple expansion to Gated Linear Units (GLU) · 3.3 Connecting structure, behavior, and optimization · 3.4 Hands-on lab · 3.5 Summary

What you learn in chapter 3 →

Chapter 4: Building smaller and faster LLMs with depth pruning

The introduction motivates task-aware transformer block selection by questioning whether a depth-pruning heuristic that works for general-purpose models also suits specific tasks.

4.1 Fundamentals and benefits of depth pruning · 4.1.1 How depth pruning works · 4.1.2 The benefits: memory consumption and speed · 4.2 Static block selection · 4.2.1 Removing first, last, middle blocks · 4.2.2 Removing by weight · 4.2.3 Static vs. weights · 4.3 Data-Driven block selection · 4.3.1 Using PyTorch hooks · 4.3.2 Understanding cosine similarity · 4.3.3 Analyzing block contributions with WinoGrande · 4.3.4 Choosing the blocks to discard · 4.3.5 Analysis of the benchmarks · 4.4 From paper to practice · 4.5 Hands-on lab · 4.6 Summary

What you learn in chapter 4 →

Chapter 5: Shaping model architectures via width pruning

The chapter introduction contrasts coarse depth pruning with more precise neuron-level width pruning and identifies wide GLU-based MLPs as suitable targets.

5.1 Selecting static neurons · 5.1.1 Identifying neurons for removal · 5.1.2 Reconstructing the MLP module · 5.1.3 Measuring the impact: benchmarks and trade-off · 5.2 Data-driven neuron selection · 5.2.1 Extracting the dynamic importance component · 5.2.2 Reconstructing the MLP with hybrid scores · 5.2.3 Running the calibration loop · 5.2.4 Creating domain-specialized models · 5.2.5 Measuring the specialization effect · 5.2.6 Inference performance and energy efficiency · 5.3 From paper to practice · 5.4 Hands-on lab · 5.5 Summary

What you learn in chapter 5 →

Chapter 6: Knowledge recovery through distillation

The introduction explains why pruning makes knowledge recovery necessary and why recovery begins with the choice of what to remove.

6.1 Choosing what to prune · 6.1.1 Experiment configuration · 6.1.2 Preparing the recovery dataset · 6.1.3 Selecting the best candidate · 6.2 Aligning Teacher and Student · 6.3 Recovering the knowledge · 6.3.1 The compound loss function · 6.3.2 Training loop and strategy comparison · 6.4 Advanced recovery techniques (FDD/ Skew KLD) · 6.4.1 Skew KLD: correcting distribution bias · 6.4.2 Feature Dynamics Distillation · 6.4.3 Results: when advanced techniques make a difference · 6.5 Practical guidelines for knowledge recovery · 6.5.1 Composite loss configuration · 6.5.2 When to use advanced techniques · 6.5.3 Combining with width pruning · 6.6 From paper to practice · 6.6.1 Layer selection and alignment · 6.6.2 Advanced techniques: when data is scarce · 6.7 Hands-on lab · 6.8 Summary

What you learn in chapter 6 →

Chapter 7: Model specialization

The chapter introduction explains how fine-tuning can specialize a model’s behavior, contrasts it with RAG, and motivates parameter-efficient fine-tuning.

7.1 Efficient fine-tuning · 7.1.1 The foundations of LoRA · 7.1.2 Extending LoRA with weight decomposition · 7.2 Quantization and the memory bottleneck · 7.3 Specializing with QLoRa and QDoRA · 7.3.1 Base model and clinical dataset · 7.3.2 Establishing the baseline: reliability and capabilities · 7.3.3 QLoRA training on the clinical dataset · 7.3.4 QDoRA training on the clinical dataset · 7.3.5 Evaluating the results against our goals · 7.3.6 Merging weights and dynamic adapter routing · 7.4 From paper to practice · 7.5 Hands-on lab · 7.6 Summary

What you learn in chapter 7 →

Chapter 8: Attention optimization

Introduces attention as an inference bottleneck whose costs grow with context length, motivating the chapter’s optimization work.

8.1 KV cache fundamentals and Hugging Face implementation · 8.1.1 What the KV cache is and why it matters · 8.1.2 Hugging Face experiment setup · 8.1.3 Hugging Face results analysis · 8.2 Scaling KV cache for production with vLLM · 8.2.1 vLLM experiment configuration · 8.2.2 vLLM results analysis · 8.3 Measuring attention redundancy · 8.3.1 Scoring with cosine similarity · 8.3.2 Analyzing results · 8.4 Removing attention modules · 8.4.1 Structural implications and framework compatibility · 8.4.2 Memory savings, quality trade-offs, and pruning limits · 8.5 From paper to practice · 8.5.1 Pruning strategies and evaluation at scale · 8.5.2 Beyond the benchmarks · 8.6 Hands-on lab · 8.7 Summary

What you learn in chapter 8 →

Chapter 9: Dynamic routing with Mixture of Experts

The introduction presents MoE as a way to add specialized experts to a dense model and defines the two routing strategies the chapter will compare.

9.1 Dynamic inference techniques · 9.2 Implementing and evaluating a two-expert MoE · 9.2.1 Building the MoE · 9.2.2 Testing the MoE · 9.2.3 Implementing hard routing · 9.2.4 How routing shapes model behavior · 9.3 Reusing a fine-tuned model as an expert · 9.3.1 Creating a new MoE class · 9.3.2 Transplant the expert and router-only training · 9.3.3 Fine-tuning the expert tail · 9.4 From paper to practice · 9.4.1 Sparse upcycling · 9.4.2 Mixtral of Experts · 9.5 Hands-on lab · 9.6 Summary

What you learn in chapter 9 →

Chapter 10: Exploring the transformer black box

The introduction frames transformer activations as a way to inspect model behavior and briefly explains what activation heatmaps and the logit lens reveal.

10.1 Capturing activations in transformers · 10.2 Mapping neuron activity across the model · 10.2.1 Two views of the same prompt · 10.2.2 The GLU mechanism in action · 10.3 From noise to answer across the layers · 10.4 From paper to practice · 10.5 Hands-on lab · 10.6 Summary

What you learn in chapter 10 →

Chapter 11: Precision pruning for bias

The introduction defines demographic bias, explains the basic idea of comparing prompts and adjusting a few neurons, and clarifies the limits of that technique.

11.1 Comparing activations between prompts · 11.1.1 Creating and verifying the prompt pairs · 11.1.2 Where the differences show up · 11.2 Finding and scaling down the neurons behind racial bias · 11.2.1 Measuring the activation difference · 11.2.2 Visualizing the differences across layers · 11.2.3 Selecting the top candidate neurons · 11.2.4 Intervening on the top-k neurons · 11.2.5 What changed after scaling · 11.2.6 Same bias, different scenario · 11.3 Replicating the procedure with gender bias · 11.3.1 Locating the gender neurons · 11.3.2 Not every neuron has the same influence · 11.3.3 Tracing the effect across layers · 11.4 Benchmarking the intervened models · 11.4.1 Measuring the impact on general capabilities · 11.4.2 Measuring bias with BBQ · 11.5 From paper to practice · 11.5.1 Anthropic, the starting point · 11.5.2 Fairness Pruning, one step ahead · 11.6 Hands-on lab · 11.7 Summary

What you learn in chapter 11 →

Chapter 12: Case study — replacing one agent call with a specialized SLM in production

The chapter introduces a costly agent-loop bottleneck and the use of shadow testing to reduce the risk of replacing its frontier API with a specialized SLM.

12.1 One call in the agent's loop · 12.2 Turning agent traffic into a specialization set · 12.3 From baseline to specialized SLM · 12.4 The research result, and what production has to fix · 12.5 The serving problem for a rearchitected model · 12.6 Will it load: from research artifact to servable model · 12.6.1 What the checkpoint contains · 12.6.2 Loading the unmodified model · 12.6.3 Loading a model with non-standard architecture · 12.7 Is it efficient: alignment and the KV cache · 12.7.1 The padding tax · 12.7.2 KV cache arithmetic after pruning · 12.8 Is it reliable: serving in the agent loop · 12.8.1 Making the output valid by construction · 12.8.2 A safety net for the SLM in production · 12.9 Proving self-hosting pays off · 12.10 Summary

What you learn in chapter 12 →

Chapter 13: Capstone — cascading specialists into a model family for agents

This introduction motivates using simpler models for repetitive tool-routing decisions in LLM-based agents.

13.1 Designing the model family · 13.2 Presenting the dataset · 13.2.1 Preparing the function calling data · 13.2.2 Preparing the examples for the model · 13.3 Creating the specialist with LoRA · 13.4 Creating the medium model through pruning and distillation · 13.4.1 Pruning the specialist · 13.4.2 Recovering knowledge · 13.5 Creating the smaller model · 13.6 Putting the family to work · 13.6.1 Accelerating the specialist with speculative decoding · 13.6.2 Cascaded routing with structural verification · 13.7 Hands-on lab · 13.8 Summary

What you learn in chapter 13 →

Concepts explained in this book

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.