liveBook · Manning

Freeze

As explained in Rearchitecting LLMs

Selective freezing is a training technique that keeps chosen model parameters fixed while allowing other parameters to update.

During MoE specialization, freezing keeps the general expert fixed while selected clinical experts and routers learn. This limits which model weights training can update. section 9.2.1

A practical setup:

  • Set every parameter to non-trainable. section 9.2.1
  • For each modified MoE layer, enable training for the clinical expert and router; leave the general expert fixed. section 9.2.1
  • Count trainable parameters to check the scope: this configuration trains 24.44% of the model weights. section 9.2.1
Listing 9.4 Preparing model weights
for param in model.parameters():
    param.requires_grad_(False) #A
 
for layer_idx in LAYERS_TO_MODIFY:
    for param in model.model.layers[layer_idx].mlp.expert_clinical.parameters():
        param.requires_grad_(True) #B
    for param in model.model.layers[layer_idx].mlp.router.parameters():
        param.requires_grad_(True) #B
 
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"Total parameters              : {total:,}")
print(f"Trainable (Expert 1 + router) : {trainable:,}  ({trainable/total:.2%})")

Use the parameter-freezing snippet to configure and verify which weights can update. section 9.2.1

Isolating an expert

An experiment freezes Expert 1 as well as Expert 0, then trains only Expert 2 tail layers and the router to compare routing and expert identity. section 9.5

How Rearchitecting LLMs covers it

How deep it goes
Shows parameter-level freezing and a trainable-weight count for the selected MoE components.
Examples
Uses clinical-expert training and an experiment freezing one expert while training another expert's tail layers and the router.
What it assumes
Requires an MoE model with modified layers that expose expert and router parameters.
What you can do after
Configure which expert and router parameters can update, then verify the trainable-weight fraction.
How it connects
Connects selective freezing with mixture of experts, routers, and expert specialization.

Also explained in

Compare how 5 books explain freeze →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.