A neuron pair consists of corresponding intermediate neurons in the MLP’s gate_proj and up_proj paths. Pruning scores pairs together, retains the highest-scoring indices, and uses them to resize both projections and the down_proj layer. section 5.1.2
Pruning 40%
With 8,192 intermediate neurons, pruning 40% removes 3,277 pairs and retains 4,915. The retained indices are sorted back into their original order before weights are transferred. section 5.1.2
Listing 5.1 Create new MLP layers
def prune_neuron_pairs(mlp, prune_percent):
gate_weight = mlp.gate_proj.weight.data.float() #1
up_weight = mlp.up_proj.weight.data.float()
importance_scores = compute_neuron_pair_importance(gate_weight,
up_weight) #2
original_intermediate_size = gate_weight.size(0)
num_neuron_pairs_to_prune = min(int(prune_percent *
➥original_intermediate_size),original_intermediate_size - 1) #3
k = original_intermediate_size - num_neuron_pairs_to_prune #4
_, indices_to_keep = torch.topk(importance_scores,
k,
largest=True,
sorted=True) #5
indices_to_keep = indices_to_keep.sort().values #6
new_gate_proj = nn.Linear(
mlp.gate_proj.in_features,
k,
bias=False).to(device) #7
new_up_proj = nn.Linear(
mlp.up_proj.in_features,
k,
bias=False).to(device)
new_down_proj = nn.Linear(
k,
mlp.down_proj.out_features,
bias=False).to(device)
new_gate_proj.weight.data =
➥mlp.gate_proj.weight.data[indices_to_keep, :] #8
new_up_proj.weight.data =
➥mlp.up_proj.weight.data[indices_to_keep, :]
new_down_proj.weight.data =
➥mlp.down_proj.weight.data[:, indices_to_keep] #9
return new_gate_proj, new_up_proj, new_down_proj, k
Inspect the pruning function, including pair scoring, index selection, and creation of smaller Linear layers. section 5.1.2
