5 Shaping model architectures via width pruning
This chapter covers
- Understanding width pruning trade-offs
- Creating specialized task models
- Building models using your own data
- Measuring speed, energy, and reasoning
The most direct structural pruning technique is depth pruning, which is especially useful if your main concern is inference performance. It's a very aggressive technique that removes a complete Transformer block.
A more refined technique is to target specific neurons in a Multi-Layer Perceptron (MLP) module with a Gated Linear Unit (GLU) structure, the gating mechanism we explored in chapter 3. It's a technique that can be used with precision, not just to improve performance, but, as you'll see, to change the model's personality. The ideal candidate for this type of optimization are wide models. In this chapter, we’ll use the Llama-3.2-1B model with a x4 expansion in its MLP layer.
We'll focus mostly on the advantages this technique can bring in terms of performance, but its uses go beyond that. To measure the general impact on performance, capabilities, and consumption, we'll introduce a more complete benchmarking system (whose technical details are in the appendices) that we'll use throughout the rest of the book.
We'll approach this technique from two complementary angles: