Token probability
Token probability is the probability a language model assigns to a candidate token at a particular sequence position, given its preceding context.
Build a Reasoning Model (From Scratch)
The probability assigned by the model to a candidate token at a particular position in the output sequence.
- How it is used
- Uses model confidence to score and rank proposed responses during inference-time self-refinement.
- How deep it goes
- Explains logits, softmax, indexed next-token extraction, and joint multiplication, then defers numerical stability to log probability.
- Examples
- Demonstrates PyTorch scoring with Germany-capital prompts ending in Berlin or Bridge and compares per-token and joint probabilities.
- What you can do after
- Build it from scratch with PyTorch by extracting per-token probabilities and multiplying them into a sequence score.
Read how Build a Reasoning Model (From Scratch) explains token probability →
LLM Evaluation and Alignment, The Foundational Ideas
A token probability is the probability that a language model assigns to a particular token at a given generation step.
- How it is used
- Uses model uncertainty as an online hallucination-detection signal for monitoring and alerting.
- How deep it goes
- Explains log-probability conversion and thresholding, but stops short of treating scores as standalone factual verification.
- Examples
- Contrasts probabilities for Einstein’s birth year and demonstrates token scoring with log probabilities and an uncertainty threshold.
- What you can do after
- Diagnose potentially uncertain generated tokens through an endpoint that exposes log probabilities.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains token probability →
Also covered in
What to know first
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- vocabularyA vocabulary is the finite set of distinct tokens a language-processing system can represent, often associated with identifiers.
Related concepts
- cross entropy lossmeasured by · Cross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.
- G-Evalused by · G-Eval is an LLM-based technique for evaluating generated text against natural-language criteria, often by generating evaluation steps and assigning scores.
- softmaxuses · Softmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.
- model calibrationused by · Model calibration is the assessment or adjustment of a model’s outputs, scores, or confidence so they correspond more reliably to observed outcomes or a reference standard.
Questions about token probability
What is token probability?
Token probability is the probability a language model assigns to a candidate token at a particular sequence position, given its preceding context.
Which Manning books explain token probability?
Build a Reasoning Model (From Scratch) and LLM Evaluation and Alignment, The Foundational Ideas.
What should I know before learning token probability?
Manning books that teach token probability usually introduce accuracy and vocabulary first.
What is the difference between token probability and semantic entropy?
Token probability is the probability a language model assigns to a candidate token at a particular sequence position, given its preceding context. Semantic entropy is an uncertainty measure computed from the entropy of clusters of sampled outputs grouped by meaning.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

