Top-k sampling
Top-k sampling is a decoding technique that restricts each next-token selection to the k candidates with the highest probabilities, then samples from those candidates.
Learn Generative AI with PyTorch
A decoding method that selects the next token from the K most likely candidates after truncating the distribution.
- How it is used
- Controls creativity and randomness during autoregressive text generation.
- How deep it goes
- Explains truncation, tradeoffs, renormalization, and implementation, but does not derive sampling theory.
- Examples
- Uses Python generation functions, top_k=3 continuations, seeded comparisons, and longer outputs from recurrent and GPT-2 models.
- What you can do after
- Apply it via PyTorch by filtering next-token probabilities with top_k before multinomial sampling.
Read how Learn Generative AI with PyTorch explains top-k sampling →
Spring AI in Action
Top-k is a language-model generation parameter that limits token candidates to a specified count of the highest-probability tokens.
- How it is used
- Controls language-token randomness and the number of retrieved documents used as context for question answering.
- How deep it goes
- Defines candidate or result counts and configuration trade-offs, but stops short of mathematical sampling or retrieval internals.
- Examples
- Uses ChatOptions.topK(4), spring.ai.ollama.chat.options.top-k=4, and SearchRequest.topK(6) in filtered game-rules retrieval.
- What you can do after
- Operate/configure Spring AI generation and retrieval by setting top-k globally, per client, per prompt, or on SearchRequest.
Deep Learning with Python, Third Edition
A decoding method that limits token selection to the K most probable candidates before sampling.
- How it is used
- Controls language-model decoding by limiting each generation step to a selectable set of likely tokens.
- How deep it goes
- Implements top-k decoding in Keras and compares it with temperature, greedy, random, and beam-search strategies.
- Examples
- Demonstrates K=5, K=20, and K=5 with temperature 0.5 through generated text continuations.
- What you can do after
- Apply it via Keras to generate text with adjustable candidate-set size and temperature.
Read how Deep Learning with Python, Third Edition explains top-k sampling →
Transformers in Action
Top-k sampling is a text-generation decoding technique that restricts sampling to a fixed number of the highest-probability tokens.
- How it is used
- Controls stochastic token selection during text generation, balancing coherent continuations with output diversity.
- How deep it goes
- Explains candidate filtering, post-softmax application, trade-offs, and a Transformers configuration, but does not derive the sampling algorithm mathematically.
- Examples
- Uses a maze analogy, successive two-candidate word choices, and a Hugging Face Transformers generation listing.
- What you can do after
- Apply it via Hugging Face Transformers by enabling sampling and setting the top_k generation parameter.
Build an Advanced RAG Application (From Scratch)
Top-k sampling restricts next-token selection to the k highest-probability candidate tokens before sampling among them.
- How it is used
- Deploys top-k sampling as a decoding method for controlling diversity and coherence during LLM text generation.
- How deep it goes
- Defines fixed-candidate filtering and random selection, illustrates k values, and stops short of implementation code.
- Examples
- Uses k = 1 and k > 1 comparisons plus a k = 3 continuation of “The quick brown fox.”
- What you can do after
- Apply it by choosing a candidate-set size and sampling among high-probability next-token alternatives.
Read how Build an Advanced RAG Application (From Scratch) explains top-k sampling →
Machine Learning Platform Engineering
Top-K sampling generates each token by restricting selection to the K most probable next tokens.
- How it is used
- Controls response variability in LLM applications such as creative generation, research assistance, and conversational systems.
- How deep it goes
- Defines candidate restriction and parameter effects, but does not explain sampling internals or implementation algorithms.
- Examples
- Uses top_k=1, top_k=40, and top_k=100 to illustrate deterministic, balanced, and broader candidate selection.
- What you can do after
- Apply top-k sampling by selecting a candidate-set size that matches the desired consistency, variety, and coherence.
Read how Machine Learning Platform Engineering explains top-k sampling →
Build a Text-to-Image Generator (from Scratch)
Top-k sampling draws the next token from a probability distribution restricted to the k highest-probability candidates.
- How it is used
- Controls creativity and coherence while autoregressively generating image tokens from text prompts.
- How deep it goes
- Defines candidate restriction and trade-offs, then shows settings and decoder-loop integration without deriving the sampling algorithm.
- Examples
- Uses an underwater fantasy city prompt, top_k 128, and a 256-step image-token generation loop.
- What you can do after
- Operate min-DALL-E generation by configuring top-k and tracing sampled image tokens through the decoder loop.
Read how Build a Text-to-Image Generator (from Scratch) explains top-k sampling →
Deep Learning with R, Third Edition
Top-k sampling selects the next token from only the k highest-scoring candidate tokens and assigns zero probability to the rest.
- How it is used
- Controls next-token selection during text generation by limiting which candidates can be sampled.
- How deep it goes
- Works through candidate selection, temperature interaction, and generated outputs, without deriving sampling mathematically.
- Examples
- Compares generated passages with k values of 5 and 20 and with top-k combined with temperature 0.5.
- What you can do after
- Apply it via op_top_k() to restrict candidate tokens and sample among them, optionally adjusting temperature.
Read how Deep Learning with R, Third Edition explains top-k sampling →
Build a Large Language Model (From Scratch)
A text-generation sampling method that restricts selection to the k highest-scoring next tokens and masks the rest.
- How it is used
- Controls GPT text-generation randomness by narrowing probabilistic next-token selection.
- How deep it goes
- Explains selection, masking, softmax, and integration into generation code, stopping short of formal derivation.
- Examples
- Demonstrates k=3 over nine logits and applies top_k=25 with temperature=1.4 during generation.
- What you can do after
- Apply it via PyTorch to modify GPT generation and tune diversity with top-k and temperature settings.
Read how Build a Large Language Model (From Scratch) explains top-k sampling →
Also covered in
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- softmaxSoftmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.
- promptA prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
Related concepts
- greedy algorithmequivalent to · A greedy algorithm constructs a solution incrementally by choosing the locally best available option at each step, generally without reconsidering earlier choices or optimizing the entire sequence globally.
- samplingkind of · Sampling is a procedure for selecting items from a collection or generating outcomes from a probability distribution according to a specified rule.
- nucleus samplingalternative to · Nucleus sampling selects a token from the smallest set of highest-probability candidates whose cumulative probability reaches a specified threshold.
- OpenAI APIcontrasts with · The OpenAI API is a programming interface that allows software applications to send requests to OpenAI models and services and receive responses.
- softmaxuses · Softmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.
Questions about top-k sampling
What is top-k sampling?
Top-k sampling is a decoding technique that restricts each next-token selection to the k candidates with the highest probabilities, then samples from those candidates.
Which Manning books explain top-k sampling?
Learn Generative AI with PyTorch, Spring AI in Action, Deep Learning with Python, Third Edition, Transformers in Action, Build an Advanced RAG Application (From Scratch), Machine Learning Platform Engineering, Build a Text-to-Image Generator (from Scratch), Deep Learning with R, Third Edition and 1 more.
What should I know before learning top-k sampling?
Manning books that teach top-k sampling usually introduce language model, softmax, prompt and token first.
What is the difference between top-k sampling and nucleus sampling?
Top-k sampling is a decoding technique that restricts each next-token selection to the k candidates with the highest probabilities, then samples from those candidates. Nucleus sampling selects a token from the smallest set of highest-probability candidates whose cumulative probability reaches a specified threshold.
What is top-k sampling a kind of?
Top-k sampling is a kind of sampling. Sampling is a procedure for selecting items from a collection or generating outcomes from a probability distribution according to a specified rule.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.








