liveBook · Manning

Logit

As explained in Build a DeepSeek Model (From Scratch)

A logit is an unnormalized real-valued score assigned to a possible output token before conversion to probabilities.

A logit is an unnormalized real-valued score assigned to a possible output token. The transformer produces these scores after contextual representations pass through the feed-forward network, remaining transformer blocks, final layer normalization, and an output projection. section 2.2.2

During next-token prediction, logits support a compact selection pipeline:

  • Select the last row of the logits matrix, because it represents predictions after the complete input context. section 2.2.3
  • Treat that row as one score for every vocabulary token; for the example input, the vector has shape (1, 50257). section 2.2.3
  • Apply softmax to convert the scores into probabilities, then use argmax to select the most likely token. section 2.2.3
  • Modify the final logits with temperature scaling, top-k, or top-p sampling to control generation before selection. section 2.2.3

Predicting after “The next day is”

The model may produce a logits matrix with shape (4, 50257), containing predictions for every input position. Only the final row is relevant: after conversion to probabilities, the highest-scoring token could be “bright.” section 2.2.3

Figure 2.12 The journey from the final context matrix to the logits matrix. The output layer projects each context-aware vector into a long vector of scores, one for each word in the vocabulary.

Trace how the final context matrix becomes a vocabulary-sized logits matrix through the output projection. section 2.2.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Traces logits from contextual representations through output projection to probability conversion, but does not derive projection internals.
Examples
Uses “The next day is” with a (4, 50257) logits matrix and “bright” as a possible selected token.
What it assumes
Assumes familiarity with token sequences and transformer blocks while grounding the path from context vectors to token probabilities.
What you can do after
Apply logits by extracting the final row, converting it with softmax, and selecting or sampling the next token.
How it connects
Connects logits to transformer blocks, attention, feed-forward networks, layer normalization, softmax, sampling, and KV caching.

Also explained in

Compare how 24 books explain logit →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.