liveBook · Manning

Structural optimization

As explained in Rearchitecting LLMs

Structural optimization improves a model by changing its architecture, components, or organization.

Structural optimization changes a model’s architecture to reduce computational requirements or adapt it to a task. Pruning removes components that contribute least to the objective, but the impact on performance depends on the technique and extent of restructuring. section 1.2 section 1.2.1

Pruning can operate at different scales:

  • Remove entire blocks for aggressive restructuring; this may require knowledge recovery. section 1.2.1
  • Remove a small number of neurons for more targeted changes that can preserve general capabilities. section 1.2.1

A common pipeline prunes the model, recovers lost capabilities through knowledge distillation, and optionally specializes it through fine-tuning. Some projects use only pruning and recovery. section 1.2 section 1.5

Figure 1.1 The model tailoring pipeline consists of core phases (shown with solid arrows) and optional phases (shown with dashed arrows). In the first phase, we adapt the structure to the model's objectives through pruning. Next, we recover capabilities it may have lost through knowledge distillation. Finally, we can optionally specialize the model through fine-tuning. An optional initial phase calibrates the base model, via a brief fine-tuning pass on the target dataset.

See how pruning, knowledge recovery, and optional specialization fit into the model-tailoring pipeline. section 1.2

student_model = prune_model(
    model=copy.deepcopy(teacher_model),
    pruning_type="DEPTH",
    layer_indices=[2, 5, 8],
    show_progress=True
)

Example: apply depth pruning to layers 2, 5, and 8. section 2.7

How Rearchitecting LLMs covers it

How deep it goes
Explains pruning from entire blocks to individual neurons and notes performance trade-offs, while pairing aggressive pruning with recovery.
Examples
Demonstrates depth pruning on layers 2, 5, and 8, and proposes comparing removal of first, middle, or final layers.
What you can do after
Build smaller, faster models by pruning architecture and recovering capabilities through distillation.
How it connects
Connects pruning with knowledge distillation for recovery and fine-tuning for optional specialization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.