liveBook · Manning

Chapter 2: An end-to-end rearchitecting project

What you will learn in chapter 2 of Rearchitecting LLMs

Introduces a general-purpose project to make a base model smaller and faster while preserving its capabilities through structural optimization and knowledge recovery.

After this chapter you will be able to

What it builds on

Section by section

Introduction

Introduces a general-purpose project to make a base model smaller and faster while preserving its capabilities through structural optimization and knowledge recovery.

Key terms: base model, general-purpose optimization, capabilities, Structural Optimization, Knowledge Recovery, pruned model

2.1 The rearchitecting workflow

This section outlines a practical pruning-and-recovery workflow and defines how the project will judge the resulting model.

Key terms: baseline, depth pruning, knowledge distillation, Structural Optimization, Knowledge Recovery, model performance, capabilities, inference efficiency, tokens per second, latency

2.2 Establishing the baseline

This section establishes the base model’s architecture, generation, benchmark-score, and inference-time baselines so later changes can be compared against them.

Key terms: performance baseline, lm-eval, model_evaluation, google/gemma-3-270m, parameters, Gemma3DecoderLayer, layers, sanity check, generate_text, do_sample=False

2.3 Applying depth pruning

The section removes the final two transformer blocks from a copy of a model, updates its structure and configuration, and explains the resulting parameter reduction.

Key terms: depth pruning, transformer blocks, final layers, original model, modified version, knowledge recovery, model.layers, nn.ModuleList, num_hidden_layers, parameters

2.4 Evaluating the impact of pruning

The section compares a model pruned by two layers with its baseline, showing that faster inference comes with degraded output and benchmark performance.

Key terms: coherent text, hallucinations, degradation, baseline, benchmarks, inference time, depth pruning, Winogrande, random chance

2.5 Recovering knowledge

This section shows how to prepare data and use logits-only knowledge distillation to help a depth-pruned model recover lost capabilities.

Key terms: Teacher model, Student model, depth pruning, prune_model, SlimPajama, tokenization, input_ids, attention_mask, Dataset, DataLoader

2.6 Analyzing the final result

This section assesses the first rearchitecture cycle through repeated benchmarks and a qualitative check of the recovered model’s answer about Paris.

Key terms: student model, knowledge distillation, controlled benchmarks, pruned model, recovered model, base model

2.7 Hands-on lab

2.8 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 1: Why rearchitecting LLMs matters Chapter 3: A blueprint to modern transformers →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.