liveBook · Manning

Model rearchitecting

As explained in Rearchitecting LLMs

Model rearchitecting is the deliberate modification of a model's architecture or components to meet particular requirements or constraints.

Model rearchitecting adapts a pretrained model by physically changing its architecture or components to meet deployment constraints or specialized requirements. Unlike prompt engineering or fine-tuning alone, it can remove layers, reconfigure attention, or reshape internal blocks. section 1.0 section 1.1.3

A domain-specific dataset can guide calibration and pruning decisions, then specialize the recovered model. A general dataset supports knowledge recovery after pruning. This structural approach can complement RAG by adapting how a model retrieves and uses external information. section 1.2.5 section 1.1.2

A rearchitecting workflow can include:

  • Establish baseline capabilities and inference metrics. section 2.0
  • Calibrate on relevant data, identify less useful components, and prune them. section 1.2.5
  • Recover capabilities through distillation, then specialize with LoRA when needed. section 1.2.5

Financial reports

For quarterly earnings analysis, calibrate a 7B general-purpose model on financial reports, prune components that contribute least to the task, recover general capabilities through distillation, and specialize with LoRA. The resulting model is described as smaller, faster, and more accurate for the use case. section 1.2.5

Figure 1.2 Dataset integration in the rearchitecting pipeline. The domain-specific dataset guides calibration of the base model, informs structural optimization decisions, and enables final specialization through LoRA fine-tuning. A general dataset supports Knowledge Recovery, ensuring the pruned model retains broad capabilities before domain-specific specialization. This dual approach optimizes each phase for the project’s objectives.

The figure maps how domain-specific and general datasets support different stages of the rearchitecting pipeline. section 1.2.5

How Rearchitecting LLMs covers it

How deep it goes
Explains calibration, pruning, distillation, and LoRA specialization, while deferring task-specific optimization to later work.
Examples
Uses a 7B model for quarterly earnings analysis and describes pruning layers before distillation and LoRA specialization.
What it assumes
Starts from a pretrained base model and uses domain-specific data to guide calibration and structural decisions.
What you can do after
Build smaller, faster models by pruning components and recovering capabilities through distillation.
How it connects
Connects structured pruning and knowledge distillation with fine-tuning, LoRA, and retrieval-augmented generation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.