An attention budget is the finite capacity a model distributes across the information in its context window. Adding marginally useful tokens can dilute the salience of critical instructions, evidence, and state, even when the prompt fits within the window. section 7.1
The attention budget differs from two other per-call limits. The context window is the model’s hard token ceiling; the token budget is the lower limit a system sets to control cost and latency. Attention remains a constraint within both: more context is not automatically better context. section 7.1 section 1.1.1
Too little information can produce vague answers or hallucinations; too much can introduce distracting noise, latency, and cost. Aim for the smallest relevant context that still supports the task. Placement also matters: in long prompts, information in the middle may have less influence than information near the beginning or end. Order and emphasize critical instructions and facts accordingly. section 1.1.1 section 1.3.2
Figure 7.1 Context is managed through four complementary techniques: selection chooses relevant tokens for the active prompt, writing persists state to external storage, compression condenses verbose history, and isolation divides complex tasks across separate context windows
The four context-management techniques show how a system can keep the active prompt relevant while managing state and task boundaries. section 7.1
