Top-p filtering, also called nucleus sampling, narrows token generation to the smallest group of high-probability tokens whose combined probability reaches a chosen threshold. Sampling from this group can reduce unreasonable predictions while preserving some variety. section 3.2.3
Why the choice matters
After “I love to eat,” foods such as barbecue and sushi receive high probabilities, while a car or 42 receive low probabilities. A rare choice such as “chalk” may still lead the model into a medically themed tangent about pica. section 3.2.3
Figure 3.11 We demonstrate text generation by starting with the phrase "I love to eat" and then showing that some possible completions that are foods, such as barbeque and sushi, have high proba-bilities, while a car and the number 42 have low probabilities. Weighted random selection chooses the word tacos. The generation loop is stopped when the EoS token appears.
Trace how token probabilities guide weighted selection during a generation loop, ending when the end-of-sequence token appears. section 3.2.3
