liveBook · Manning

Expansion ratio

As explained in Rearchitecting LLMs

An expansion rate is the ratio of a module's intermediate dimension to the model's hidden-state dimension, expressed as a percentage or factor.

The expansion ratio, or expansion rate, sets an MLP’s intermediate width relative to the model’s hidden-state width: intermediate size ÷ hidden-state size. It can be expressed as a factor or percentage. In width pruning, specifying the target ratio sets the final intermediate size directly, rather than specifying a pruning percentage. section 13.4.1

A lower ratio can reduce model size and computational cost, but the effect on task performance depends on the model. Experiments with Llama-3.2-1B and 3B found that some lower ratios could improve performance on certain tasks; experiments with the Gemma family gave different results. section 3.2.2

From 300% to 225%

With a hidden-state size of 1024, an intermediate size of 3072 gives an expansion rate of 300%. A target of 225% gives 2304 neurons: a 25% reduction in intermediate size, while the hidden-state size stays fixed. section 13.4.1

For a GLU MLP, gate_proj and up_proj work in pairs. Pruning must evaluate neuron pairs jointly and remove corresponding neurons from both projections so their element-wise multiplication remains valid. section 3.2.2

Listing 13.13 Width pruning
EXPANSION_RATE = 225              
EXPANSION_DIVISOR = 128        
   
model_m_pruned, width_stats = opf.prune_model(
    model=copy.deepcopy(depth_pruned_model),
    pruning_type="MLP_GLU",
    neuron_selection_method="PPM", #1    
    pruning_percentage=None, #A           
    expansion_rate=EXPANSION_RATE, #2
    expansion_divisor=EXPANSION_DIVISOR, #3
    dataloader=calibration_dataloader, #B
    show_progress=True,
    return_stats=True,
)
model_m_pruned.generation_config.pad_token_id = tokenizer.pad_token_id
model_m_pruned.eval()
 
inter = [layer.mlp.gate_proj.out_features for layer in model_m_pruned.model.layers]
print(f"layers: {len(inter)}")
print(f"intermediate sizes: {Counter(inter)}")

This width-pruning workflow sets expansion_rate to 225 and expansion_divisor to 128, then checks gate_proj output widths across the pruned model. The divisor makes the intermediate size a multiple of 128, which usually makes better use of the GPU. section 13.4.1

Validate both architecture and task performance after pruning. In one specialist cascade, all 25 blocks reached an intermediate size of 2304. After depth and width pruning, valid JSON fell from 100% to 74.1%, and exact match fell from 96.3% to 51.9% against the original specialist. section 13.4.1

How Rearchitecting LLMs covers it

How deep it goes
Defines the ratio, calculates a target width, and checks a pruning result; model-dependent experiments do not establish a universal optimum.
Examples
Calculates 225% as 2304 neurons for a 1024-wide hidden state and compares pruned specialist performance with its original.
What you can do after
Apply it via OptiPFair to target and validate a GLU MLP’s pruned intermediate width.
How it connects
Connects expansion ratio to MLPs, gated linear units, width pruning, calibration, and GPU utilization.

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.