Top-k sampling
Top-k sampling is a decoding technique that restricts each next-token selection to the k candidates with the highest probabilities, then samples from those candidates.
Learn Generative AI with PyTorch
A decoding method that selects the next token from the K most likely candidates after truncating the distribution.
- How it is used
- Controls creativity and randomness during autoregressive text generation.
- How deep it goes
- Explains truncation, tradeoffs, renormalization, and implementation, but does not derive sampling theory.
- Examples
- Uses Python generation functions, top_k=3 continuations, seeded comparisons, and longer outputs from recurrent and GPT-2 models.
- What you can do after
- Apply it via PyTorch by filtering next-token probabilities with top_k before multinomial sampling.
Read how Learn Generative AI with PyTorch explains top-k sampling →
Spring AI in Action
Top-k is a language-model generation parameter that limits token candidates to a specified count of the highest-probability tokens.
- How it is used
- Controls language-token randomness and the number of retrieved documents used as context for question answering.
- How deep it goes
- Defines candidate or result counts and configuration trade-offs, but stops short of mathematical sampling or retrieval internals.
- Examples
- Uses ChatOptions.topK(4), spring.ai.ollama.chat.options.top-k=4, and SearchRequest.topK(6) in filtered game-rules retrieval.
- What you can do after
- Operate/configure Spring AI generation and retrieval by setting top-k globally, per client, per prompt, or on SearchRequest.
Deep Learning with Python, Third Edition
A decoding method that limits token selection to the K most probable candidates before sampling.
- How it is used
- Controls language-model decoding by limiting each generation step to a selectable set of likely tokens.
- How deep it goes
- Implements top-k decoding in Keras and compares it with temperature, greedy, random, and beam-search strategies.
- Examples
- Demonstrates K=5, K=20, and K=5 with temperature 0.5 through generated text continuations.
- What you can do after
- Apply it via Keras to generate text with adjustable candidate-set size and temperature.
Read how Deep Learning with Python, Third Edition explains top-k sampling →
Transformers in Action
Top-k sampling is a text-generation decoding technique that restricts sampling to a fixed number of the highest-probability tokens.
- How it is used
- Controls stochastic token selection during text generation, balancing coherent continuations with output diversity.
- How deep it goes
- Explains candidate filtering, post-softmax application, trade-offs, and a Transformers configuration, but does not derive the sampling algorithm mathematically.
- Examples
- Uses a maze analogy, successive two-candidate word choices, and a Hugging Face Transformers generation listing.
- What you can do after
- Apply it via Hugging Face Transformers by enabling sampling and setting the top_k generation parameter.
Machine Learning Platform Engineering
Top-K sampling generates each token by restricting selection to the K most probable next tokens.
- How it is used
- Controls response variability in LLM applications such as creative generation, research assistance, and conversational systems.
- How deep it goes
- Defines candidate restriction and parameter effects, but does not explain sampling internals or implementation algorithms.
- Examples
- Uses top_k=1, top_k=40, and top_k=100 to illustrate deterministic, balanced, and broader candidate selection.
- What you can do after
- Apply top-k sampling by selecting a candidate-set size that matches the desired consistency, variety, and coherence.
Read how Machine Learning Platform Engineering explains top-k sampling →
Build a Text-to-Image Generator (from Scratch)
Top-k sampling draws the next token from a probability distribution restricted to the k highest-probability candidates.
- How it is used
- Controls creativity and coherence while autoregressively generating image tokens from text prompts.
- How deep it goes
- Defines candidate restriction and trade-offs, then shows settings and decoder-loop integration without deriving the sampling algorithm.
- Examples
- Uses an underwater fantasy city prompt, top_k 128, and a 256-step image-token generation loop.
- What you can do after
- Operate min-DALL-E generation by configuring top-k and tracing sampled image tokens through the decoder loop.
Read how Build a Text-to-Image Generator (from Scratch) explains top-k sampling →
Deep Learning with R, Third Edition
Top-k sampling selects the next token from only the k highest-scoring candidate tokens and assigns zero probability to the rest.
- How it is used
- Controls next-token selection during text generation by limiting which candidates can be sampled.
- How deep it goes
- Works through candidate selection, temperature interaction, and generated outputs, without deriving sampling mathematically.
- Examples
- Compares generated passages with k values of 5 and 20 and with top-k combined with temperature 0.5.
- What you can do after
- Apply it via op_top_k() to restrict candidate tokens and sample among them, optionally adjusting temperature.
Read how Deep Learning with R, Third Edition explains top-k sampling →
Build a Large Language Model (From Scratch)
A text-generation sampling method that restricts selection to the k highest-scoring next tokens and masks the rest.
- How it is used
- Controls GPT text-generation randomness by narrowing probabilistic next-token selection.
- How deep it goes
- Explains selection, masking, softmax, and integration into generation code, stopping short of formal derivation.
- Examples
- Demonstrates k=3 over nine logits and applies top_k=25 with temperature=1.4 during generation.
- What you can do after
- Apply it via PyTorch to modify GPT generation and tune diversity with top-k and temperature settings.
Read how Build a Large Language Model (From Scratch) explains top-k sampling →
Related concepts
- greedy algorithmequivalent to · A greedy algorithm builds a solution incrementally by choosing the locally best available option at each step, without generally accounting for all future consequences or revising prior choices.
- samplingkind of · Sampling is a procedure for selecting or generating observations, values, or outcomes from a larger population, signal, or probability distribution according to a specified method.
- top-p filteringalternative to · Top-p filtering is a token-sampling technique that selects from the smallest probability-ranked set of candidates whose cumulative probability reaches a specified threshold.
- argmaxequivalent to · Argmax is an operation that identifies an argument, position, or index at which a function or collection attains its maximum value.
- softmaxuses · Softmax is a function that exponentiates a vector of scores and normalizes the results into nonnegative values that sum to one.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.