Top-p filtering
Top-p filtering is a token-sampling technique that selects from the smallest probability-ranked set of candidates whose cumulative probability reaches a specified threshold.
Build Python Web Apps with Streamlit
top_p is a language-model sampling parameter that limits token selection to the smallest set whose cumulative probability reaches a specified threshold.
- How it is used
- Uses top_p to adjust response variability when generating multiple trivia questions.
- How deep it goes
- Defines the cumulative-probability cutoff and demonstrates an API argument, without developing sampling mathematics.
- Examples
- Shows a Python LLM wrapper and a trivia-game call using temperature=0.7 and top_p=0.9.
- What you can do after
- Apply it via OpenAI chat completions to adjust question variability when the selected model supports explicit top_p settings.
Read how Build Python Web Apps with Streamlit explains top-p filtering →
Build a Reasoning Model (From Scratch)
A decoding method that restricts sampling to the smallest set of tokens whose cumulative probability reaches a chosen threshold p.
- How it is used
- Controls inference-time sampling so generated candidates remain diverse without admitting very low-probability tokens.
- How deep it goes
- Explains sorting, cumulative probabilities, cutoff variants, masking, scattering, renormalization, and generation integration in PyTorch.
- Examples
- Uses ten-token toy probabilities, p=0.8 plots and tensors, full-vocabulary logits, and city-token sampling outputs.
- What you can do after
- Apply it via PyTorch by inserting top_p_filter between probability conversion and torch.multinomial sampling.
Read how Build a Reasoning Model (From Scratch) explains top-p filtering →
Natural Language Processing in Action, Second Edition
Nucleus sampling restricts decoding to the smallest set of tokens whose cumulative probability reaches a specified threshold before sampling.
- How it is used
- Controls probabilistic token selection when generating text with a generative language model.
- How deep it goes
- Defines adaptive candidate selection and demonstrates implementation, but defers detailed coverage of other sampling techniques.
- Examples
- Uses Hugging Face GPT-2 to generate continuations for “NLP is a” with top_p=0.92.
- What you can do after
- Apply it via Hugging Face generation arguments to produce probabilistically varied continuations.
Read how Natural Language Processing in Action, Second Edition explains top-p filtering →
LLMs in Production
Top-p sampling restricts token selection to the smallest candidate set whose cumulative probability reaches a specified threshold.
- How it is used
- Controls language-model generation by restricting next-token sampling to a probability mass chosen for the desired rigidity or flexibility.
- How deep it goes
- Explains cumulative filtering, contrasts fixed-count top-k filtering, and shows implementation placement after temperature but stops short of derivation.
- Examples
- Uses a woman-or-man probability example, threshold values of 0.05 and 0.95, and a top-k versus top-p sampling exercise.
- What you can do after
- Apply it via language-model generation code and compare top-p with top-k, argmax, and multinomial sampling.
Spring AI in Action
Top-p is a language-model generation parameter from 0 to 1 that limits token candidates to the smallest probability set whose cumulative probability reaches the parameter value.
- How it is used
- Controls how many high-probability tokens remain eligible during chat response generation.
- How deep it goes
- Defines the cumulative-probability rule and configuration paths, but stops short of mathematical implementation details.
- Examples
- Uses ranked candidate words with probabilities and Spring Boot or ChatOptions configuration examples.
- What you can do after
- Configure it through Spring Boot properties or ChatOptions when creating or invoking a chat client.
Machine Learning Platform Engineering
Top-P sampling generates each token from the smallest set of candidates whose cumulative probability reaches a specified threshold.
- How it is used
- Controls response diversity in generative and conversational LLM applications.
- How deep it goes
- Defines dynamic cumulative-probability selection and contrasts it with fixed-size Top-K sampling, without detailing implementation internals.
- Examples
- Uses Top-P settings of 0.1, 0.9, and 1.0 in a parameter listing.
- What you can do after
- Apply it via sampling configuration to balance response variety and coherence for a chosen application.
Read how Machine Learning Platform Engineering explains top-p filtering →
Building Reliable AI Systems
Top-p sampling is a language-model generation setting that samples from the smallest set of tokens whose cumulative probability reaches a specified threshold.
- How it is used
- Controls the candidate-token pool during LLM generation to balance response diversity, focus, and reliability.
- How deep it goes
- Defines the cumulative-probability threshold and compares two settings in a chatbot scenario, without providing API code or a tuning procedure.
- Examples
- Contrasts wireless-headphone recommendations at top-p 0.95 and 0.5 to show the difference between detail and focus.
- What you can do after
- Operate/configure it by adjusting top-p to narrow or broaden candidate tokens for a desired response style.
Read how Building Reliable AI Systems explains top-p filtering →
How Large Language Models Work
Nucleus sampling is a token-generation method that samples only from the smallest set of highest-probability tokens whose cumulative probability reaches a selected threshold.
- How it is used
- Controls token selection to limit rare or unreasonable outputs while retaining probabilistic variation.
- How deep it goes
- Defines the selection strategy and its motivation but does not explain threshold tuning or implementation details.
- Examples
- Uses food completions after “I love to eat” and the unusual “chalk” continuation into pica.
- What you can do after
- Evaluate generation risks by checking which sampling algorithm an LLM uses.
Read how How Large Language Models Work explains top-p filtering →
Generative AI in Action
Nucleus sampling is a text-generation technique that samples from the smallest dynamically selected set of tokens whose cumulative probability exceeds a specified threshold.
- How it is used
- Controls randomness and token selection in language-model text-generation APIs.
- How deep it goes
- Explains cumulative-probability selection, changing group size, trade-offs, and configuration guidance without mathematical derivation.
- Examples
- Uses a top_p value of 0.9 and contrasts lower and higher thresholds through token-choice and output-quality effects.
- What you can do after
- Apply it via a language-model API and tune top_p for more predictable or more varied generated text.
AI Agents in Action, Second Edition
Top-p is a generation parameter that restricts sampling to the smallest set of tokens whose cumulative probability reaches a specified threshold.
- How it is used
- Controls next-token variety during generation by excluding low-probability candidates from the sampling pool.
- How deep it goes
- Explains cumulative-probability filtering, renormalization, sampling, tuning ranges, and interactions, without descending into model internals.
- Examples
- Uses coder-agent and creative-writer settings to illustrate how generation controls support consistency or creative variation.
- What you can do after
- Configure generation variety by selecting a tighter or freer candidate set and avoiding simultaneous temperature and top-p tuning.
Read how AI Agents in Action, Second Edition explains top-p filtering →
Transformers in Action
Top-p sampling generates each token by sampling from the smallest candidate set whose cumulative probability reaches a specified threshold.
- How it is used
- Controls stochastic text generation by limiting sampling to an adaptive set of high-probability tokens.
- How deep it goes
- Defines cumulative-probability selection, contrasts top-k, explains confidence-dependent variation, and demonstrates implementation.
- Examples
- Uses a 0.9 movie-or-pie example and a Hugging Face model.generate configuration.
- What you can do after
- Operate/configure Hugging Face generation with do_sample and top_p to produce diverse, controlled text.
Data Analysis with LLMs
A top_p parameter is a model-generation setting that limits eligible output tokens to a cumulative probability mass.
- How it is used
- Controls output-token randomization during language-model generation.
- How deep it goes
- Explains probability-mass selection, value range, and randomization effects without detailing token-scoring internals.
- Examples
- Uses multiple generated SQL queries for one prompt as a postprocessing choice.
- What you can do after
- Operate/configure OpenAI model generation by setting top_p between 0 and 1 as an alternative to temperature.
Related concepts
- samplingkind of · Sampling is a procedure for selecting or generating observations, values, or outcomes from a larger population, signal, or probability distribution according to a specified method.
- top-k samplingalternative to · Top-k sampling is a decoding technique that restricts each next-token selection to the k candidates with the highest probabilities, then samples from those candidates.
- temperaturecontrasts with · A temperature parameter adjusts the concentration of a probability distribution, affecting sampling randomness or the likelihood of accepting less favorable transitions.
- language modelconfigures · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.