Routing lets a mixture-of-experts (MoE) layer use only a subset of its expert networks for each token. A small learnable router scores every expert; top-k selection chooses which experts run, avoiding computation for the rest. section 4.2.3 section 4.2.2
The router can be a linear layer. For input X with shape (tokens, embedding dimension) and learned routing matrix W with shape (embedding dimension, experts), the logits XW have shape (tokens, experts). Each entry is a raw suitability score for one token–expert pair. section 4.2.3
Four tokens, three experts
Four 8-dimensional token embeddings form a (4, 8) input matrix. Multiplying by an (8, 3) routing matrix produces a (4, 3) score matrix. With k=2, each row selects two experts; the third expert does not process that token. section 4.2.3 section 4.2.2
Figure 4.7 The routing mechanism. The input matrix is multiplied by a learned routing matrix to produce an expert selector matrix, which contains a raw score for each expert for each token.

Follow the matrix multiplication from token embeddings to per-expert logits. section 4.2.3
A routed forward pass can be implemented in four steps:
- Flatten a batch of token embeddings and calculate router logits with a learned linear projection. section 4.5
- Add the current per-expert balancing bias, select the top k logits per token, and apply softmax to those selected logits to obtain gate weights. section 6.2.1
- For each expert, gather the tokens that selected it, run the expert, and accumulate its weighted outputs at the corresponding token positions. section 4.5 section 6.2.1
- Add the routed result to the shared-expert result. Shared experts process every token; routed experts process only selected tokens. section 4.5 section 6.2.1
Listing 6.12 The forward method of DeepSeekMoE (Routing and Dispatch)
def forward(self, x: torch.Tensor):
B, S, D = x.shape
x_flat = x.reshape(-1, D)
# Shared expert path (dense)
shared_out = torch.zeros_like(x)
for exp in self.shared_experts: shared_out += exp(x)
# --- Routed expert path (sparse) —
router_logits = self.gate(x_flat) #A
router_logits_with_bias = router_logits + self.bias.to(router_logits.dtype) #B
top_k_logits, top_k_indices = torch.topk(router_logits_with_bias, self.top_k, dim=-1)
gates = F.softmax(top_k_logits, dim=-1, dtype=torch.float).type_as(x) #C
routed_out_flat = torch.zeros_like(x_flat)
for i in range(self.n_routed): #C
mask = (top_k_indices == i)
row_idx, which_k = torch.where(mask)
if row_idx.numel() == 0:
continue
w = gates[row_idx, which_k].unsqueeze(-1)
exp_in = x_flat[row_idx]
exp_out = self.routed_experts[i](exp_in)
routed_out_flat.index_add_(0, row_idx, exp_out * w) # ... (bias update and final output comes next)Inspect the PyTorch routing and dispatch path, including tensor reshaping, top-k selection, and gate weights. section 6.2.1
Sparse routing can become imbalanced. Experts selected early receive more training and may become increasingly favored, while rarely selected experts get few or no updates. Overused experts can also become bottlenecks. section 4.3
One balancing approach adds an auxiliary loss, but its weight trades balance against the main training objective. Auxiliary-loss-free balancing instead adjusts a non-trainable per-expert bias: increase it for underloaded experts and decrease it for overloaded ones, influencing the next top-k selection. section 4.4.4
Listing 4.5 The update_bias Method for Load Balancing
@torch.no_grad()
def update_bias(self, x: torch.Tensor):
# Call once per optimizer step on the same tokens seen by forward.
# Uses the SAME logits (including current bias) to estimate loads.
N = x.shape[0] * x.shape[1]
logits = F.linear(x.reshape(-1,
self.d_model), self.centroids) + self.bias #A
_, idx = torch.topk(logits, self.top_k, dim=-1)
counts = torch.bincount(idx.flatten(),
minlength=self.n_routed).float() #B
avg = counts.sum() / max(1, self.n_routed)
# Smooth, bounded update; avoids large jumps from sign()
# violation > 0 => under-loaded (we want to increase its prior)
violation = (avg - counts) / (avg + 1e-6) #C
self.bias.add_(self.bias_lr * torch.tanh(violation)) #DSee a no-gradient bias update that counts selections, compares them with average load, and applies a bounded adjustment. section 4.5

































