A logit is an unnormalized score produced by a model before softmax converts scores into probabilities. In language generation, one logit corresponds to each possible next token. section 8.5.2 section 11.2.4
Logits connect model outputs to token selection in several steps:
- A recurrent or Transformer model produces one score for every vocabulary token. section 8.5.2 section 11.2.4
- Softmax converts the scores into a probability distribution used for sampling. section 8.5.2 section 11.2.4
- Temperature modifies the logits before softmax: lower values favor predictable choices, while higher values increase variation. section 8.5.2
- Top-K sampling filters the resulting distribution to a selected number of high-probability tokens. section 8.5.2
Sampling the next word
Given the prompt “I'm not going to see,” the model produces logits for candidate continuations. With temperature 1 and no top-K limit, repeated samples vary among several words. With temperature 0.5 and top_k=3, outputs concentrate on choices such as “you,” “the,” and “her.” Raising temperature to 2 produces more unusual continuations and can reduce coherence.
