Top-p filtering controls model randomness by limiting eligible output tokens to a cumulative probability mass. Tokens are considered from highest to lowest probability until that mass exceeds top_p; the next token is selected from the eligible set. section 3.5.3
Use top_p with these practical guidelines:
- Choose a value between 0 and 1; higher values admit lower-probability tokens and produce more randomization. section 3.5.3
- Treat top_p as an alternative to temperature rather than changing both parameters in one model invocation. section 3.5.3
- Generate multiple completions when randomness makes several answers useful, then select the preferred result through postprocessing. section 3.5.3
Comparing generated SQL queries
For one input prompt, generate multiple SQL queries with randomized settings and choose the preferred query during postprocessing. section 3.5.3
