Top-P filtering, also called nucleus sampling, generates each token from the smallest candidate set whose cumulative probability reaches a configured threshold. Unlike Top-K sampling, it adapts vocabulary size to model confidence: confident predictions consider fewer tokens, while uncertain predictions consider more. This controlled randomness supports varied responses without admitting the full tail of unlikely candidates. section 12.1.1
Use the sampling threshold according to the response goal:
- Favor deterministic, consistent behavior for factual retrieval by using temperature 0; Top-P is part of the broader randomness configuration. section 12.1.1
- Allow controlled variation for creative generation, brainstorming, research assistance, or conversational responses. section 12.1.1
Listing 12.1 Parameters that control output randomness
temperature=0.0 #1 temperature=0.7 #2 temperature=1.5 #3 top_k=1 #4 top_k=40 #5 top_k=100 #6 top_p=0.1 #7 top_p=0.9 #8 top_p=1.0 #9
Compare sample Top-P configurations of 0.1, 0.9, and 1.0 alongside temperature and Top-K settings. section 12.1.1
