liveBook · Manning

Feature importance score

As explained in Build a DeepSeek Model (From Scratch)

Expert importance is the aggregate selection weight assigned to an expert across the tokens in a batch.

An expert’s importance score is the sum of its selector weights across all tokens in a batch. In the expert selector matrix, tokens occupy rows and experts occupy columns, so each score is a column sum. section 4.3.1

Figure 4.14 Calculating Expert Importance by summing the probabilities down each column.

The figure shows how summing each expert’s column gives its batch importance score. section 4.3.1

Importance is not token load

Scores of 1.4, 1.0, and 1.6 indicate unequal importance. But equal scores can conceal unequal workloads: one expert may receive one token at probability 1.0 while another receives four at 0.25 each. section 4.3.1 section 4.3.2

Importance scores feed the coefficient-of-variation penalty and, when divided by the batch’s token count, give each expert’s mean router probability. They measure selection weight, not dispatched-token count. section 4.3.1 section 4.3.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines the batch column sum, works through numerical scores, and connects them to coefficient of variation and mean router probability.
Examples
Calculates scores of 1.4, 1.0, and 1.6, then contrasts equal scores from one high-probability token and four low-probability tokens.
What it assumes
Defines the selector matrix’s token rows and expert columns before summing weights, grounding the metric in batch routing.
What you can do after
Evaluate expert balance by aggregate routing weight while recognizing that equal importance scores do not guarantee equal token workloads.
How it connects
Connects expert importance to coefficient of variation, auxiliary loss, router probability, and token load.

Also explained in

Compare how 10 books explain feature importance score →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.