liveBook · Manning

Logits matrix

As explained in Build a DeepSeek Model (From Scratch)

A logits matrix is a matrix of routing scores assigning each token a score for each available expert.

A logits matrix stores raw, unnormalized scores before routing or token selection. In a mixture-of-experts router, multiplying token embeddings by a learned routing matrix produces one score for every token–expert pair. section 4.2.3

Token-to-expert scores

With four token embeddings of shape (4, 8) and three experts, a routing matrix of shape (8, 3) produces a logits matrix of shape (4, 3). Each row represents a token, each column an expert, and each cell its raw suitability score. Top-k routing keeps the highest two scores per row, masks the rest with negative infinity, and applies row-wise softmax to obtain weights that sum to 1. section 4.2.3 section 4.2.4

The same structure also appears during autoregressive language-model inference:

  • Select the final row because it contains predictions for the next token after the complete input; earlier rows are unnecessary for that decision. section 2.2.3
  • Apply softmax to convert the final logits vector into vocabulary probabilities, then use argmax or controlled sampling to choose a token. section 2.2.3
Figure 4.7 The routing mechanism. The input matrix is multiplied by a learned routing matrix to produce an expert selector matrix, which contains a raw score for each expert for each token.

Visualize the matrix multiplication that turns token representations into expert-specific routing scores. section 4.2.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines matrix shapes, score interpretation, masking, softmax conversion, and last-row extraction, but stops short of implementation code.
Examples
Uses a four-token, three-expert matrix and a four-position, 50,257-vocabulary inference tensor.
What it assumes
Assumes token embeddings, matrix multiplication, experts, vocabulary probabilities, and softmax are familiar enough to follow tensor shapes.
What you can do after
Apply it to interpret expert routing scores or extract and normalize the final vector for next-token selection.
How it connects
Connects routers, mixture-of-experts, top-k selection, softmax, KV cache, MQA, GQA, and controlled generation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.