A routing weight is a scalar that controls an expert’s contribution to a mixture-of-experts output. In soft routing, two or more experts combine their outputs; a larger weight indicates a stronger contribution for that token. section 9.2.2
In a two-expert block, a linear router projects each token’s hidden state to two logits. Softmax converts them into weights that sum to 1. The softmax runs in FP32 even when the rest of the model uses float16. section 9.2.1
During training, Gaussian noise is added to the logits before softmax. This encourages exploration of both experts and helps prevent early routing collapse, when a favored expert receives more gradient and the other receives little learning signal. The original MLP is retained as a frozen general expert; a trainable copy can specialize. section 9.2.1
Listing 9.1 Trainable2ExpertMoE
class Trainable2ExpertMoE(nn.Module):
def __init__(self, original_mlp, hidden_size):
super().__init__()
target_dtype = next(original_mlp.parameters()).dtype
target_device = next(original_mlp.parameters()).device
self.expert_general = original_mlp #A
self.expert_clinical = copy.deepcopy(original_mlp) #B
self.router = nn.Linear(hidden_size, 2,
bias=False,
dtype=target_dtype,
device=target_device) #C
self.last_routing = None
def forward(self, hidden_states):
router_logits = self.router(hidden_states)
if self.training:
noise = torch.randn_like(router_logits) * 0.35 #1
router_logits = router_logits + noise
routing_weights = F.softmax(
router_logits.float(), dim=-1
).to(hidden_states.dtype) #2
out_general = self.expert_general(hidden_states)
out_clinical = self.expert_clinical(hidden_states)
# Soft routing: each token receives a weighted blend of both experts
output = out_general * routing_weights[..., 0].unsqueeze(-1)
[CA]+ out_clinical * routing_weights[..., 1].unsqueeze(-1) #3
self.last_routing = routing_weights.detach() #D
return output
Inspect the two-expert block’s router initialization and noisy routing calculation. section 9.2.1
To inspect routing, run a forward pass and read the selected block’s stored last_routing values. Pair the weights with token strings to compare preferences across prompts and layers. section 9.2.2
Listing 9.7 Analyze routing
def analyze_routing(text, layer_index=0):
inputs = tokenizer(text, return_tensors="pt").to(DEVICE)
with torch.no_grad():
with torch.amp.autocast("cuda", dtype=torch.float16):
_ = model(**inputs)
moe_layer = model.model.layers[layer_index].mlp #A
routing_weights = moe_layer.last_routing.squeeze(0) #B
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
print(f"Routing analysis (layer {layer_index}):")
print(f"{'Token':<18} | {'Expert 0 (General)':<20} | {'Expert 1 (Clinical)'}")
print("-" * 65)
for token, weights in zip(tokens, routing_weights):
clean = token.replace("Ġ", "").replace(" ", "")
print(f"{clean:<18} | {weights[0].item():<20.4f} | {weights[1].item():.4f}")
Use the routing analysis function to print each token’s general and clinical expert weights for a selected layer. section 9.2.2
General and clinical prompts
At layer 23, a general question about Barcelona shows a mild preference for the general expert. In a clinical prompt, the clinical expert’s weight reaches about 0.62 on “with,” although preferences vary by token. section 9.2.2
Running both experts achieves specialization but activates all available expert weights. Selective activation with more experts offers a different efficiency trade-off: only experts considered most relevant to a token run on a given forward pass. section 9.2.2
