liveBook · Manning

Gaussian noise

As explained in Rearchitecting LLMs

Gaussian noise is a random perturbation whose values follow a normal probability distribution.

During Mixture of Experts training, Gaussian noise is added to router logits before softmax. This encourages early exploration, reducing the risk that routing collapses onto one expert before another can specialize. section 9.2.1 section 9.4.1

Listing 9.1 Trainable2ExpertMoE
class Trainable2ExpertMoE(nn.Module):
    def __init__(self, original_mlp, hidden_size):
        super().__init__()
        target_dtype = next(original_mlp.parameters()).dtype
        target_device = next(original_mlp.parameters()).device
 
        self.expert_general = original_mlp #A
        self.expert_clinical = copy.deepcopy(original_mlp) #B
        self.router = nn.Linear(hidden_size, 2, 
                                bias=False,
                                dtype=target_dtype, 
                                device=target_device) #C
        self.last_routing = None 
 
    def forward(self, hidden_states):
        router_logits = self.router(hidden_states)
 
        if self.training:
            noise = torch.randn_like(router_logits) * 0.35  #1
            router_logits = router_logits + noise
 
        routing_weights = F.softmax(
            router_logits.float(), dim=-1
        ).to(hidden_states.dtype)  #2
 
        out_general = self.expert_general(hidden_states)
        out_clinical = self.expert_clinical(hidden_states)
 
        # Soft routing: each token receives a weighted blend of both experts
        output = out_general * routing_weights[..., 0].unsqueeze(-1) 
[CA]+ out_clinical * routing_weights[..., 1].unsqueeze(-1) #3
        self.last_routing = routing_weights.detach() #D
 
        return output

Inspect the two-expert implementation: it adds 0.35-scaled random noise to router logits during training. section 9.2.1

How Rearchitecting LLMs covers it

How deep it goes
Implements scaled Gaussian samples added to router logits only during training, using a scale of 0.35.
Examples
Shows 0.35-scaled noise on a two-expert router to help a clinical expert specialize while preserving a frozen general expert.
What it assumes
Assumes an MoE setup with router logits and expert assignments; explains how exploration helps when experts begin with identical weights.
What you can do after
Apply it by adding 0.35-scaled torch.randn_like noise to router logits during training.
How it connects
Connects router-logit perturbation to Mixture of Experts routing, softmax probabilities, and expert specialization.

Also explained in

Compare how 4 books explain Gaussian noise →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.