liveBook · Manning

Hard routing

As explained in Rearchitecting LLMs

Hard routing is a routing behavior that assigns a token to selected experts without continuously weighting all candidate expert outputs.

Hard routing selects experts for each token rather than continuously mixing all candidate expert outputs. In a two-expert mixture of experts (MoE), the router makes a binary choice and only the selected expert processes the token. This selective activation offers a path to greater total model capacity without inference cost growing in the same proportion, although it does not by itself establish a measured speedup. section 9.2.2 section 9.2.3

The router acts on contextual hidden states, so the same token can be assigned differently in different contexts. Soft and hard routing can use the same trained expert and router weights: switching the inference forward pass requires no second training run. section 9.6 section 9.2.3

A two-expert hard inference pass can be implemented as follows:

  • Compute router logits from hidden states and take argmax over the expert dimension to choose expert 0 or 1 for each token. section 9.2.3
  • Allocate an output tensor with the same shape as the hidden states. Build one Boolean mask for each expert; each token belongs to exactly one mask. section 9.2.3
  • For each nonempty mask, run the corresponding expert on its selected hidden states and place the results back at their original positions. section 9.2.3
Listing 9.8 Trainable2ExpertMoE hard inference
class Trainable2ExpertMoE(nn.Module):
    def __init__(self, original_mlp, hidden_size):
        ... #A
 
    def forward(self, hidden_states):
        router_logits = self.router(hidden_states)
        routing_weights = F.softmax(
            router_logits.float(), 
            dim=-1).to(hidden_states.dtype)
 
        if self.training:
         .... #A
        else:
            expert_choice = torch.argmax(
                router_logits, 
                dim=-1) #B
 
            output = torch.zeros_like(hidden_states)
 
            mask_general = (expert_choice == 0)
            mask_clinical = (expert_choice == 1)
 
            if mask_general.any():
                output[mask_general] = self.expert_general(
                    hidden_states[mask_general]) #C
            if mask_clinical.any():
                output[mask_clinical] = self.expert_clinical(
                    hidden_states[mask_clinical]) #C
 
            hard_decisions = F.one_hot(expert_choice, num_classes=2).float()
            self.last_routing = hard_decisions.detach()
 
        return output

The evaluation branch uses argmax, masks, and indexed assignment to route tokens to two experts. section 9.2.3

To replace existing MoE blocks, construct blocks with the hard-routing forward method and copy the trained clinical expert and router state into them. The general expert is initialized from the existing general expert. Inspect per-token routing afterward to verify that the decisions are one-hot. section 9.2.3

General and clinical prompts

In a layer-23 inspection, a general-knowledge prompt routed every displayed token to the general expert. A clinical prompt routed a mixture of tokens to both experts, including “chronic” to the clinical expert. These are individual routing observations, not a guarantee that every clinical term selects the clinical expert. section 9.2.3

Routing can also change generated behavior. On one general-knowledge prompt, the hard model produced a shorter answer than the base and soft models; this single prompt is only a control test. Across five general benchmarks, both MoE variants improved over the base in four and declined slightly on PIQA. The small movements do not support definitive conclusions. section 9.2.4

How Rearchitecting LLMs covers it

How deep it goes
Works through argmax selection, Boolean masks, expert execution, output placement, and replacement of trained blocks.
Examples
Inspects routing for general-knowledge and clinical prompts and compares generated answers and general benchmarks against soft routing and the base model.
What you can do after
Apply hard routing by modifying a trained two-expert PyTorch forward pass and inspecting per-token choices.
How it connects
Connects hard routing to soft routing, mixture of experts, contextual hidden states, and sparse upcycling.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.