liveBook · Manning

Nucleus sampling

Also called top-p

Nucleus sampling selects a token from the smallest set of highest-probability candidates whose cumulative probability reaches a specified threshold.

How 14 Manning books explain it

Build Python Web Apps with Streamlit

top_p is a language-model sampling parameter that limits token selection to the smallest set whose cumulative probability reaches a specified threshold.

How it is used
Controls response variability when generating multiple trivia questions.
How deep it goes
Explains cumulative-probability filtering and demonstrates passing the setting through an API wrapper.
Examples
Uses a multi-question trivia game with a seed, previously asked questions, and top_p=0.9.
What you can do after
Apply it via the OpenAI Python client to configure a supported model’s response variability.

Read how Build Python Web Apps with Streamlit explains nucleus sampling →

Build a Reasoning Model (From Scratch)

A decoding method that restricts sampling to the smallest set of tokens whose cumulative probability reaches a chosen threshold p.

How it is used
Controls inference-time sampling so generated candidates remain diverse without admitting very low-probability tokens.
How deep it goes
Explains sorting, cumulative probabilities, cutoff variants, masking, scattering, renormalization, and generation integration in PyTorch.
Examples
Uses ten-token toy probabilities, p=0.8 plots and tensors, full-vocabulary logits, and city-token sampling outputs.
What you can do after
Apply it via PyTorch by inserting top_p_filter between probability conversion and torch.multinomial sampling.

Read how Build a Reasoning Model (From Scratch) explains nucleus sampling →

Natural Language Processing in Action, Second Edition

Nucleus sampling restricts decoding to the smallest set of tokens whose cumulative probability reaches a specified threshold before sampling.

How it is used
Controls probabilistic token selection when generating text with a generative language model.
How deep it goes
Defines adaptive candidate selection and demonstrates implementation, but defers detailed coverage of other sampling techniques.
Examples
Uses Hugging Face GPT-2 to generate continuations for “NLP is a” with top_p=0.92.
What you can do after
Apply it via Hugging Face generation arguments to produce probabilistically varied continuations.

Read how Natural Language Processing in Action, Second Edition explains nucleus sampling →

LLMs in Production

Top-p sampling restricts token selection to the smallest candidate set whose cumulative probability reaches a specified threshold.

How it is used
Controls language-model generation by restricting next-token sampling to a probability mass chosen for the desired rigidity or flexibility.
How deep it goes
Explains cumulative filtering, contrasts fixed-count top-k filtering, and shows implementation placement after temperature but stops short of derivation.
Examples
Uses a woman-or-man probability example, threshold values of 0.05 and 0.95, and a top-k versus top-p sampling exercise.
What you can do after
Apply it via language-model generation code and compare top-p with top-k, argmax, and multinomial sampling.

Read how LLMs in Production explains nucleus sampling →

Spring AI in Action

Top-p is a language-model generation parameter from 0 to 1 that limits token candidates to the smallest probability set whose cumulative probability reaches the parameter value.

How it is used
Controls how many high-probability tokens remain eligible during chat response generation.
How deep it goes
Defines the cumulative-probability rule and configuration paths, but stops short of mathematical implementation details.
Examples
Uses ranked candidate words with probabilities and Spring Boot or ChatOptions configuration examples.
What you can do after
Configure it through Spring Boot properties or ChatOptions when creating or invoking a chat client.

Read how Spring AI in Action explains nucleus sampling →

Build an Advanced RAG Application (From Scratch)

Top-p, or nucleus, sampling selects from the smallest probability-ranked token set whose cumulative probability reaches a specified threshold.

How it is used
Controls randomness, coherence, diversity, and style during autoregressive language-model text generation.
How deep it goes
Defines cumulative-probability selection, contrasts top-k trade-offs, and illustrates one toy continuation without implementation code.
Examples
Uses p = 0.8 with “The quick brown fox” and visualizes probability-mass filtering in Figure 5.9.
What you can do after
Apply it via a decoder to adjust candidate-set flexibility and randomness when generating diverse, coherent text.

Read how Build an Advanced RAG Application (From Scratch) explains nucleus sampling →

Machine Learning Platform Engineering

Top-P sampling generates each token from the smallest set of candidates whose cumulative probability reaches a specified threshold.

How it is used
Controls response diversity in generative and conversational LLM applications.
How deep it goes
Defines dynamic cumulative-probability selection and contrasts it with fixed-size Top-K sampling, without detailing implementation internals.
Examples
Uses Top-P settings of 0.1, 0.9, and 1.0 in a parameter listing.
What you can do after
Apply it via sampling configuration to balance response variety and coherence for a chosen application.

Read how Machine Learning Platform Engineering explains nucleus sampling →

Building Reliable AI Systems

Top-p sampling is a language-model generation setting that samples from the smallest set of tokens whose cumulative probability reaches a specified threshold.

How it is used
Tunes candidate-token diversity to support more relevant and reliable language-model responses.
How deep it goes
Explains cumulative-probability filtering at each generation step and the effect of changing the threshold.
Examples
Illustrates high and low thresholds with an e-commerce chatbot recommending wireless headphones.
What you can do after
Evaluate how top-p settings influence the breadth and focus of generated responses.

Read how Building Reliable AI Systems explains nucleus sampling →

How Large Language Models Work

Nucleus sampling is a token-generation method that samples only from the smallest set of highest-probability tokens whose cumulative probability reaches a selected threshold.

How it is used
Controls token selection to limit rare or unreasonable outputs while retaining probabilistic variation.
How deep it goes
Defines the selection strategy and its motivation but does not explain threshold tuning or implementation details.
Examples
Uses food completions after “I love to eat” and the unusual “chalk” continuation into pica.
What you can do after
Evaluate generation risks by checking which sampling algorithm an LLM uses.

Read how How Large Language Models Work explains nucleus sampling →

Applied Reinforcement Learning

Nucleus sampling draws a token from the smallest set of highest-probability tokens whose cumulative probability reaches a chosen threshold.

How it is used
Uses stochastic token selection during PPO rollout generation to collect varied responses for exploration.
How deep it goes
Defines cumulative-probability filtering and illustrates a threshold, without detailing its implementation.
Examples
Uses top_p = 0.9 and candidate sets ranging from three to fifty tokens to illustrate uncertainty-dependent selection.
What you can do after
Apply top_p = 0.9 during rollout generation to vary responses while excluding the implausible long tail.

Read how Applied Reinforcement Learning explains nucleus sampling →

Generative AI in Action

Nucleus sampling is a text-generation technique that samples from the smallest dynamically selected set of tokens whose cumulative probability exceeds a specified threshold.

How it is used
Controls randomness and token selection in language-model text-generation APIs.
How deep it goes
Explains cumulative-probability selection, changing group size, trade-offs, and configuration guidance without mathematical derivation.
Examples
Uses a top_p value of 0.9 and contrasts lower and higher thresholds through token-choice and output-quality effects.
What you can do after
Apply it via a language-model API and tune top_p for more predictable or more varied generated text.

Read how Generative AI in Action explains nucleus sampling →

AI Agents in Action, Second Edition

Top-p is a generation parameter that restricts sampling to the smallest set of tokens whose cumulative probability reaches a specified threshold.

How it is used
Controls next-token variety during generation by excluding low-probability candidates from the sampling pool.
How deep it goes
Explains cumulative-probability filtering, renormalization, sampling, tuning ranges, and interactions, without descending into model internals.
Examples
Uses coder-agent and creative-writer settings to illustrate how generation controls support consistency or creative variation.
What you can do after
Configure generation variety by selecting a tighter or freer candidate set and avoiding simultaneous temperature and top-p tuning.

Read how AI Agents in Action, Second Edition explains nucleus sampling →

Transformers in Action

Top-p sampling generates each token by sampling from the smallest candidate set whose cumulative probability reaches a specified threshold.

How it is used
Controls token selection during text generation to balance output diversity and coherence.
How deep it goes
Explains post-softmax cumulative-threshold filtering, contrasts adaptive and fixed candidate sets, and demonstrates library-level configuration.
Examples
Uses a 0.9 threshold illustration and a Transformers generation call with three returned sequences.
What you can do after
Apply top-p through Hugging Face Transformers’ model.generate() with do_sample and top_p.

Read how Transformers in Action explains nucleus sampling →

Data Analysis with LLMs

A top_p parameter is a model-generation setting that limits eligible output tokens to a cumulative probability mass.

How it is used
Controls output-token randomization during language-model generation.
How deep it goes
Explains probability-mass selection, value range, and randomization effects without detailing token-scoring internals.
Examples
Uses multiple generated SQL queries for one prompt as a postprocessing choice.
What you can do after
Operate/configure OpenAI model generation by setting top_p between 0 and 1 as an alternative to temperature.

Read how Data Analysis with LLMs explains nucleus sampling →

Also covered in

What to know first

Related concepts

Questions about nucleus sampling

What is nucleus sampling?

Nucleus sampling selects a token from the smallest set of highest-probability candidates whose cumulative probability reaches a specified threshold.

Which Manning books explain nucleus sampling?

Build Python Web Apps with Streamlit, Build a Reasoning Model (From Scratch), Natural Language Processing in Action, Second Edition, LLMs in Production, Spring AI in Action, Build an Advanced RAG Application (From Scratch), Machine Learning Platform Engineering, Building Reliable AI Systems and 6 more.

What should I know before learning nucleus sampling?

Manning books that teach nucleus sampling usually introduce language model, prompt, token and probability theory first.

What is the difference between nucleus sampling and temperature?

Nucleus sampling selects a token from the smallest set of highest-probability candidates whose cumulative probability reaches a specified threshold. A temperature parameter controls the spread of a probability distribution or the likelihood of selecting less-favored alternatives in stochastic sampling or search.

What is nucleus sampling a kind of?

Nucleus sampling is a kind of sampling. Sampling is a procedure for selecting items from a collection or generating outcomes from a probability distribution according to a specified rule.

What is nucleus sampling also called?

Nucleus sampling is also called top-p.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.