liveBook · Manning

Torch.topk

As explained in Rearchitecting LLMs

torch.topk is a PyTorch operation that returns the largest or smallest k elements of a tensor together with their indices.

torch.topk selects k tensor entries by value and returns their indices. It supports neuron retention during pruning and expert selection during inference-time routing. section 5.1.2 section 9.3.1

The selected indices serve different purposes in each workflow:

  • For pruning, select the k highest-importance neuron pairs, then sort their indices to restore the neurons’ original order before copying weights. section 5.1.2
  • For inference routing, select each token’s top_k expert logits, normalize those selected values with softmax, and use the indices to assign tokens to experts. section 9.3.1
Listing 5.1 Create new MLP layers
def prune_neuron_pairs(mlp, prune_percent):
   
    gate_weight = mlp.gate_proj.weight.data.float() #1
    up_weight = mlp.up_proj.weight.data.float() 
 
    importance_scores = compute_neuron_pair_importance(gate_weight, 
                      up_weight) #2 
    original_intermediate_size = gate_weight.size(0) 
    
    num_neuron_pairs_to_prune = min(int(prune_percent * 
➥original_intermediate_size),original_intermediate_size - 1) #3
    
    k = original_intermediate_size - num_neuron_pairs_to_prune #4
 
    _, indices_to_keep = torch.topk(importance_scores, 
                k, 
                largest=True, 
                sorted=True) #5
 
    indices_to_keep = indices_to_keep.sort().values #6
 
    new_gate_proj = nn.Linear(
                mlp.gate_proj.in_features, 
                k, 
                bias=False).to(device) #7
    new_up_proj = nn.Linear(
                mlp.up_proj.in_features, 
                k, 
                bias=False).to(device)
    new_down_proj = nn.Linear(
                k, 
                mlp.down_proj.out_features, 
                bias=False).to(device) 
 
    new_gate_proj.weight.data =
➥mlp.gate_proj.weight.data[indices_to_keep, :] #8
    new_up_proj.weight.data =
➥mlp.up_proj.weight.data[indices_to_keep, :] 
    new_down_proj.weight.data = 
➥mlp.down_proj.weight.data[:, indices_to_keep] #9
 
    return new_gate_proj, new_up_proj, new_down_proj, k

Inspect the pruning function’s top-k call and the subsequent index sort used to preserve neuron order. section 5.1.2

How Rearchitecting LLMs covers it

How deep it goes
Demonstrates k-based selection, index reordering for pruning, and selected-logit normalization for routing.
Examples
Uses 40% pruning of 8,192 neurons and four-expert top_k=2 routing to illustrate selection.
What you can do after
Apply PyTorch's torch.topk to retain high-importance neurons or choose top-scoring experts per token.
How it connects
Links tensor ranking to neural network pruning, importance scores, router logits, softmax, and mixture-of-experts routing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.