Top-k
Top-k is a selection operation that retains the k items ranked highest according to a specified value or criterion.
CUDA for LLMs
Top-k selection identifies the k highest values in a collection and optionally their indices.
- How it is used
- Uses top-k to route each token to a limited set of mixture-of-experts networks.
- How deep it goes
- Develops sequential selection, sorted-buffer and heap implementations, then compares their GPU parallelism with reduction and production sorting approaches.
- Examples
- Works through top-2 selection from eight expert scores and shows PyTorch and CUDA routing paths.
- What you can do after
- Builds a CUDA top-k scan from scratch and evaluates its synchronization and numerical tradeoffs in MoE routing.
Build a DeepSeek Model (From Scratch)
Top-k selection retains the k highest-scoring experts for each token and discards the others.
- How it is used
- Uses sparse routing to reduce per-token computation in Mixture-of-Experts models.
- How deep it goes
- Explains row-wise selection, negative-infinity masking, and softmax weighting, but stops short of implementing expert-output merging.
- Examples
- Uses three experts, k=2, the token "The," scores 1, 5, 4, and 60%/40% normalized weights.
- What you can do after
- Apply it via a router to select experts and compute normalized weights for each token.
Read how Build a DeepSeek Model (From Scratch) explains top-k →
Transformers in Action
Top-k selection is a selection operation that retains the k highest-scoring candidates.
- How it is used
- Routes tokens to a few experts and narrows retrieved documents to the most relevant candidates.
- How deep it goes
- Defines selection and weighting with executable gating logic, then extends it to capacity, balancing, and randomized routing.
- Examples
- Demonstrates PyTorch sparse gating, GShard-style top-2 routing, and RAG retrieval refined with Flashrank or JinaRerank.
- What you can do after
- Apply it via torch.topk, sparse MoE gates, or retrieval pipelines that rerank and compress selected context.
Also covered in
- Graph Algorithms for Data Science
- Grokking AI Applications
- LLMs in Production
- Reinforcement Learning from Human Feedback
- Context EngineeringIn early access (MEAP)
- Event Processing in Action
- Build Applications with Local AI Models on a Mac
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- MapA map is an associative data structure that stores key-value associations and supports retrieving values by key.
- information retrievalInformation retrieval is the process of finding and selecting information relevant to a query or information need from a collection.
Related concepts
- expert routingused by · Expert routing is the mechanism that selects which expert subnetworks process an input or token in a mixture-of-experts model.
- insertion sortuses · Insertion sort is a comparison-sorting algorithm that builds a sorted sequence by inserting each successive element into its proper position among the elements already sorted.
- min-heapuses · A min-heap is a heap-ordered tree in which each parent has a value or priority no greater than its children, placing a minimum element at the root.
Questions about top-k
What is top-k?
Top-k is a selection operation that retains the k items ranked highest according to a specified value or criterion.
Which Manning books explain top-k?
CUDA for LLMs, Build a DeepSeek Model (From Scratch) and Transformers in Action.
What should I know before learning top-k?
Manning books that teach top-k usually introduce language model, fine-tuning, Map and information retrieval first.
What is the difference between top-k and top-p?
Top-k is a selection operation that retains the k items ranked highest according to a specified value or criterion. Top-p is a language-model generation parameter that restricts token selection to the smallest set of likely tokens whose cumulative probability reaches a specified threshold.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


