liveBook · Manning

Neuron pair

As explained in Rearchitecting LLMs

A neuron pair is the corresponding pair of intermediate MLP neurons represented across the gated projection paths.

A neuron pair consists of corresponding intermediate neurons in the MLP’s gate_proj and up_proj paths. Pruning scores pairs together, retains the highest-scoring indices, and uses them to resize both projections and the down_proj layer. section 5.1.2

Pruning 40%

With 8,192 intermediate neurons, pruning 40% removes 3,277 pairs and retains 4,915. The retained indices are sorted back into their original order before weights are transferred. section 5.1.2

Listing 5.1 Create new MLP layers
def prune_neuron_pairs(mlp, prune_percent):
   
    gate_weight = mlp.gate_proj.weight.data.float() #1
    up_weight = mlp.up_proj.weight.data.float() 
 
    importance_scores = compute_neuron_pair_importance(gate_weight, 
                      up_weight) #2 
    original_intermediate_size = gate_weight.size(0) 
    
    num_neuron_pairs_to_prune = min(int(prune_percent * 
➥original_intermediate_size),original_intermediate_size - 1) #3
    
    k = original_intermediate_size - num_neuron_pairs_to_prune #4
 
    _, indices_to_keep = torch.topk(importance_scores, 
                k, 
                largest=True, 
                sorted=True) #5
 
    indices_to_keep = indices_to_keep.sort().values #6
 
    new_gate_proj = nn.Linear(
                mlp.gate_proj.in_features, 
                k, 
                bias=False).to(device) #7
    new_up_proj = nn.Linear(
                mlp.up_proj.in_features, 
                k, 
                bias=False).to(device)
    new_down_proj = nn.Linear(
                k, 
                mlp.down_proj.out_features, 
                bias=False).to(device) 
 
    new_gate_proj.weight.data =
➥mlp.gate_proj.weight.data[indices_to_keep, :] #8
    new_up_proj.weight.data =
➥mlp.up_proj.weight.data[indices_to_keep, :] 
    new_down_proj.weight.data = 
➥mlp.down_proj.weight.data[:, indices_to_keep] #9
 
    return new_gate_proj, new_up_proj, new_down_proj, k

Inspect the pruning function, including pair scoring, index selection, and creation of smaller Linear layers. section 5.1.2

How Rearchitecting LLMs covers it

How deep it goes
Explains importance-based selection, index reordering, and projection resizing, but does not specify how pair importance is computed.
Examples
Shows 40% pruning of 8,192 intermediate neurons, retaining 4,915 pairs in original index order.
What you can do after
Apply pair selection to resize gated MLP projections while preserving retained neurons’ original order.
How it connects
Connects MLPs with gate projection, up projection, down projection, and knowledge distillation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.