A token limit is a configured ceiling on how many tokens a language model may generate in one response. Think of it as a response-length guardrail: the model can stop at the limit even when more output would otherwise follow. section 2.1.3
Unlike temperature and top-p, which influence which token the model selects, a token limit does not shape sampling. It limits response length, helps control expense, and must fit alongside prompt tokens within the model’s context window. section 2.1.3 section 2.6
Set the limit with these practical checks:
- Measure tokens rather than estimating from character or text length; token count and text length are not equivalent. section 2.6
- Set the ceiling just above the expected response length to reduce unexpectedly long outputs and unnecessary expense. section 2.1.3
- Reserve room for the prompt because prompt tokens plus the configured maximum must remain within the context window. section 2.1.3
- Use per-run token budgets and alerting when an agent workflow, such as research, needs an overall spending boundary. section 11.4
A deep-research run
A research planner may search, extract, analyze, critique, and write. A per-run token budget places a boundary around that execution, while the response limit controls the length of an individual model response. Caching repeated queries and throttling web or API calls provide additional cost controls. section 11.4
Table 2.1 LLM parameters and how they can be used to control output
| Parameter |
Typical range |
What it really does |
When to raise / lower |
|---|---|---|---|
| Temperature |
0–2 |
Scales every token’s probability before sampling. Higher = flatter distribution ⇒ more randomness/creativity. Lower = sharper ⇒ more deterministic, repetitive output. |
Raise for brainstorming, fiction. Lower for consistent answers, code. |
| top-p (nucleus) |
0–1 |
Keeps only the smallest set of tokens whose cumulative probability ≥ p; renormalizes, then samples. Controls variety without letting low-probability “tail” tokens in. |
Lower(< 0.9) for tight, on-topic prose. Higher (~1) for free-form generation. |
| max_tokens |
1–8192 + (model-dependent) |
A hard ceiling on how many tokens the model may generate in the response. (Prompt tokens + max ≤ context window.) |
Set just above the expected length. It prevents rambling and keeps cost down. |
| presence_penalty |
–2 … +2 |
Adds a penalty once a token has appeared; encourages new topics |
Use 0.6–1.0 to avoid repeating already-mentioned concepts. |
| frequency_penalty |
–2 … +2 |
Penalizes tokens in proportion to their current count; discourages phrase repetition |
Use 0.2–0.8 to reduce echoing or loops in long answers. |
Use the parameter table to compare max_tokens with temperature, top-p, and penalty terms, including their roles and adjustment guidance. section 2.1.3
