Top-p filtering restricts sampling to the smallest set of next-token candidates whose cumulative probability reaches a chosen threshold. The model renormalizes that set, then samples from it, excluding the low-probability tail. section 2.1.3 section 2.1.1
Use this mental model when choosing generation controls:
- Lower top-p values, such as below 0.9, keep prose tighter and more on topic; values near 1 allow more free-form generation. section 2.1.3
- Temperature changes the shape of the entire probability distribution, while top-p removes its least-likely tail before sampling. section 2.1.3
- Max tokens limits response length rather than selecting candidate tokens, so it serves a different purpose. section 2.1.3
- Adjust either temperature or top-p when tuning randomness; changing both can produce unpredictable interactions. section 2.1.3
Choosing a setting by role
A strict coder agent benefits from consistent generation controls, while a creative writer agent benefits from settings that permit more variation. Top-p supplies one way to tighten or widen the candidate set, but should be tuned alongside the agent’s role rather than treated as a strict guarantee. section 2.1.3
Table 2.1 LLM parameters and how they can be used to control output
| Parameter |
Typical range |
What it really does |
When to raise / lower |
|---|---|---|---|
| Temperature |
0–2 |
Scales every token’s probability before sampling. Higher = flatter distribution ⇒ more randomness/creativity. Lower = sharper ⇒ more deterministic, repetitive output. |
Raise for brainstorming, fiction. Lower for consistent answers, code. |
| top-p (nucleus) |
0–1 |
Keeps only the smallest set of tokens whose cumulative probability ≥ p; renormalizes, then samples. Controls variety without letting low-probability “tail” tokens in. |
Lower(< 0.9) for tight, on-topic prose. Higher (~1) for free-form generation. |
| max_tokens |
1–8192 + (model-dependent) |
A hard ceiling on how many tokens the model may generate in the response. (Prompt tokens + max ≤ context window.) |
Set just above the expected length. It prevents rambling and keeps cost down. |
| presence_penalty |
–2 … +2 |
Adds a penalty once a token has appeared; encourages new topics |
Use 0.6–1.0 to avoid repeating already-mentioned concepts. |
| frequency_penalty |
–2 … +2 |
Penalizes tokens in proportion to their current count; discourages phrase repetition |
Use 0.2–0.8 to reduce echoing or loops in long answers. |
Use the parameter table to compare top-p with temperature, max_tokens, and penalty settings by range, effect, and tuning guidance. section 2.1.3
