Top-p filtering, also called nucleus sampling, controls generation randomness by selecting the smallest group of likely next tokens whose cumulative probability reaches a configured threshold. The group changes for each prediction, so the technique adapts to the model’s probability distribution. section 2.8.5 section 3.2.5
Use the threshold to shape the balance between consistency and variety:
- Set top_p to a lower value to consider fewer tokens, producing more consistent and less creative text. section 3.2.5
- Set top_p to a higher value to allow more token choices, increasing creative variation while also increasing errors and randomness. section 3.2.5
- Change either temperature or top_p for a generation configuration rather than adjusting both. section 3.2.5
Filtering the probability tail
With top_p set to 0.9, generation samples from tokens comprising 90% of the probability distribution and excludes the remaining 10%, which may contain highly random choices and nonsensical hallucinations. A typical starting range is 0.7 to 0.95, adjusted for the use case. section 3.2.5
Figure 2.11 Example showing how top-p works
Visualizes how the selected nucleus changes as top-p filtering determines which tokens remain eligible. section 2.8.5
