torch.topk selects k tensor entries by value and returns their indices. It supports neuron retention during pruning and expert selection during inference-time routing. section 5.1.2 section 9.3.1
The selected indices serve different purposes in each workflow:
- For pruning, select the k highest-importance neuron pairs, then sort their indices to restore the neurons’ original order before copying weights. section 5.1.2
- For inference routing, select each token’s top_k expert logits, normalize those selected values with softmax, and use the indices to assign tokens to experts. section 9.3.1
Listing 5.1 Create new MLP layers
def prune_neuron_pairs(mlp, prune_percent):
gate_weight = mlp.gate_proj.weight.data.float() #1
up_weight = mlp.up_proj.weight.data.float()
importance_scores = compute_neuron_pair_importance(gate_weight,
up_weight) #2
original_intermediate_size = gate_weight.size(0)
num_neuron_pairs_to_prune = min(int(prune_percent *
➥original_intermediate_size),original_intermediate_size - 1) #3
k = original_intermediate_size - num_neuron_pairs_to_prune #4
_, indices_to_keep = torch.topk(importance_scores,
k,
largest=True,
sorted=True) #5
indices_to_keep = indices_to_keep.sort().values #6
new_gate_proj = nn.Linear(
mlp.gate_proj.in_features,
k,
bias=False).to(device) #7
new_up_proj = nn.Linear(
mlp.up_proj.in_features,
k,
bias=False).to(device)
new_down_proj = nn.Linear(
k,
mlp.down_proj.out_features,
bias=False).to(device)
new_gate_proj.weight.data =
➥mlp.gate_proj.weight.data[indices_to_keep, :] #8
new_up_proj.weight.data =
➥mlp.up_proj.weight.data[indices_to_keep, :]
new_down_proj.weight.data =
➥mlp.down_proj.weight.data[:, indices_to_keep] #9
return new_gate_proj, new_up_proj, new_down_proj, k
Inspect the pruning function’s top-k call and the subsequent index sort used to preserve neuron order. section 5.1.2
