liveBook · Manning

Standard moe

As explained in Build a DeepSeek Model (From Scratch)

Standard MoE is the book's baseline mixture-of-experts model using conventional auxiliary-loss-based load balancing.

Standard MoE is a baseline mixture-of-experts architecture that uses a conventional auxiliary load-balancing loss. A matched training comparison configured it with approximately 101.30 million parameters. section 4.6

The empirical comparison highlights:

  • Training on TinyStories for 5,000 iterations produced a best validation loss of 1.9854. section 4.6
  • The baseline required 14.29 minutes and reached 350.0 iterations per minute. section 4.6
  • Expert-selection frequencies were uneven in a sample batch, revealing imbalanced routing. section 4.6
Figure 4.29 Expert selection frequency for the Standard-MoE model in a sample batch. The uneven distribution highlights the problem of imbalanced routing.

The expert-selection frequency plot makes the baseline’s uneven routing visible and provides a direct contrast with dynamic balancing. section 4.6

Matched comparison

Against DeepSeek-MoE, Standard MoE achieved a best validation loss of 1.9854, compared with 1.9451, while training more slowly and reaching lower throughput. section 4.6

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines conventional load balancing and reports empirical outcomes, but does not detail the routing implementation in the supplied treatment.
Examples
Uses TinyStories training, 5,000 iterations, validation-loss curves, and expert-selection frequency plots.
What you can do after
Evaluate mixture-of-experts designs using validation loss, training time, throughput, and parameter-matched comparisons.
How it connects
Connects mixture-of-experts routing, auxiliary load-balancing loss, validation loss, and DeepSeek-MoE.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.