liveBook · Manning

Token probability

Token probability is the probability a language model assigns to a candidate token at a particular sequence position, given its preceding context.

How 2 Manning books explain it

Build a Reasoning Model (From Scratch)

The probability assigned by the model to a candidate token at a particular position in the output sequence.

How it is used
Uses model confidence to score and rank proposed responses during inference-time self-refinement.
How deep it goes
Explains logits, softmax, indexed next-token extraction, and joint multiplication, then defers numerical stability to log probability.
Examples
Demonstrates PyTorch scoring with Germany-capital prompts ending in Berlin or Bridge and compares per-token and joint probabilities.
What you can do after
Build it from scratch with PyTorch by extracting per-token probabilities and multiplying them into a sequence score.

Read how Build a Reasoning Model (From Scratch) explains token probability →

LLM Evaluation and Alignment, The Foundational Ideas

A token probability is the probability that a language model assigns to a particular token at a given generation step.

How it is used
Uses model uncertainty as an online hallucination-detection signal for monitoring and alerting.
How deep it goes
Explains log-probability conversion and thresholding, but stops short of treating scores as standalone factual verification.
Examples
Contrasts probabilities for Einstein’s birth year and demonstrates token scoring with log probabilities and an uncertainty threshold.
What you can do after
Diagnose potentially uncertain generated tokens through an endpoint that exposes log probabilities.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains token probability →

Also covered in

What to know first

Related concepts

Questions about token probability

What is token probability?

Token probability is the probability a language model assigns to a candidate token at a particular sequence position, given its preceding context.

Which Manning books explain token probability?

Build a Reasoning Model (From Scratch) and LLM Evaluation and Alignment, The Foundational Ideas.

What should I know before learning token probability?

Manning books that teach token probability usually introduce accuracy and vocabulary first.

What is the difference between token probability and semantic entropy?

Token probability is the probability a language model assigns to a candidate token at a particular sequence position, given its preceding context. Semantic entropy is an uncertainty measure computed from the entropy of clusters of sampled outputs grouped by meaning.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.