liveBook · Manning

Routing weight

As explained in Rearchitecting LLMs

A routing weight is a scalar produced by a mixture-of-experts router that controls an expert's contribution to the combined output.

A routing weight is a scalar that controls an expert’s contribution to a mixture-of-experts output. In soft routing, two or more experts combine their outputs; a larger weight indicates a stronger contribution for that token. section 9.2.2

In a two-expert block, a linear router projects each token’s hidden state to two logits. Softmax converts them into weights that sum to 1. The softmax runs in FP32 even when the rest of the model uses float16. section 9.2.1

During training, Gaussian noise is added to the logits before softmax. This encourages exploration of both experts and helps prevent early routing collapse, when a favored expert receives more gradient and the other receives little learning signal. The original MLP is retained as a frozen general expert; a trainable copy can specialize. section 9.2.1

Listing 9.1 Trainable2ExpertMoE
class Trainable2ExpertMoE(nn.Module):
    def __init__(self, original_mlp, hidden_size):
        super().__init__()
        target_dtype = next(original_mlp.parameters()).dtype
        target_device = next(original_mlp.parameters()).device
 
        self.expert_general = original_mlp #A
        self.expert_clinical = copy.deepcopy(original_mlp) #B
        self.router = nn.Linear(hidden_size, 2, 
                                bias=False,
                                dtype=target_dtype, 
                                device=target_device) #C
        self.last_routing = None 
 
    def forward(self, hidden_states):
        router_logits = self.router(hidden_states)
 
        if self.training:
            noise = torch.randn_like(router_logits) * 0.35  #1
            router_logits = router_logits + noise
 
        routing_weights = F.softmax(
            router_logits.float(), dim=-1
        ).to(hidden_states.dtype)  #2
 
        out_general = self.expert_general(hidden_states)
        out_clinical = self.expert_clinical(hidden_states)
 
        # Soft routing: each token receives a weighted blend of both experts
        output = out_general * routing_weights[..., 0].unsqueeze(-1) 
[CA]+ out_clinical * routing_weights[..., 1].unsqueeze(-1) #3
        self.last_routing = routing_weights.detach() #D
 
        return output

Inspect the two-expert block’s router initialization and noisy routing calculation. section 9.2.1

To inspect routing, run a forward pass and read the selected block’s stored last_routing values. Pair the weights with token strings to compare preferences across prompts and layers. section 9.2.2

Listing 9.7 Analyze routing
def analyze_routing(text, layer_index=0):
    inputs = tokenizer(text, return_tensors="pt").to(DEVICE)
    with torch.no_grad():
        with torch.amp.autocast("cuda", dtype=torch.float16):
            _ = model(**inputs)
 
    moe_layer      = model.model.layers[layer_index].mlp #A
    routing_weights = moe_layer.last_routing.squeeze(0) #B
    tokens          = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
 
    print(f"Routing analysis (layer {layer_index}):")
    print(f"{'Token':<18} | {'Expert 0 (General)':<20} | {'Expert 1 (Clinical)'}")
    print("-" * 65)
    for token, weights in zip(tokens, routing_weights):
        clean = token.replace("Ġ", "").replace(" ", "")
        print(f"{clean:<18} | {weights[0].item():<20.4f} | {weights[1].item():.4f}")

Use the routing analysis function to print each token’s general and clinical expert weights for a selected layer. section 9.2.2

General and clinical prompts

At layer 23, a general question about Barcelona shows a mild preference for the general expert. In a clinical prompt, the clinical expert’s weight reaches about 0.62 on “with,” although preferences vary by token. section 9.2.2

Running both experts achieves specialization but activates all available expert weights. Selective activation with more experts offers a different efficiency trade-off: only experts considered most relevant to a token run on a given forward pass. section 9.2.2

How Rearchitecting LLMs covers it

How deep it goes
Explains router logits, noisy training, FP32 softmax, and token-level inspection in a two-expert implementation.
Examples
Compares token weights for general and clinical prompts and proposes freezing and two-phase training experiments.
What you can do after
Diagnose routing preferences by inspecting token-level weights across prompts and selected layers.
How it connects
Connects mixture-of-experts routing with softmax, expert specialization, gradient flow, and selective activation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.