liveBook · Manning

Attention budget

As explained in Context Engineering

An attention budget is the finite capacity a model has to distribute attention across the information it receives.

An attention budget is the finite capacity a model distributes across the information in its context window. Adding marginally useful tokens can dilute the salience of critical instructions, evidence, and state, even when the prompt fits within the window. section 7.1

The attention budget differs from two other per-call limits. The context window is the model’s hard token ceiling; the token budget is the lower limit a system sets to control cost and latency. Attention remains a constraint within both: more context is not automatically better context. section 7.1 section 1.1.1

Too little information can produce vague answers or hallucinations; too much can introduce distracting noise, latency, and cost. Aim for the smallest relevant context that still supports the task. Placement also matters: in long prompts, information in the middle may have less influence than information near the beginning or end. Order and emphasize critical instructions and facts accordingly. section 1.1.1 section 1.3.2

Figure 7.1 Context is managed through four complementary techniques: selection chooses relevant tokens for the active prompt, writing persists state to external storage, compression condenses verbose history, and isolation divides complex tasks across separate context windows
context management v6

The four context-management techniques show how a system can keep the active prompt relevant while managing state and task boundaries. section 7.1

How Context Engineering covers it

How deep it goes
Defines the constraint and explains salience and placement consequences without specifying a quantitative budget.
Examples
Uses an order-support assistant and long prompts with potentially neglected middle content.
What you can do after
Evaluate/understand how context quantity and placement affect salience, cost, and latency.
How it connects
Connects context windows and token budgets with context management and the lost-in-the-middle effect.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Context Engineering. The section links point to the parts of the book it draws on.