Perception selects, prioritizes, and assembles information for an agent’s next reasoning step. It is the entry point to the agent’s view of the world: evidence filtered out or buried in noise may never influence the resulting plan or action. Memory can supply prior context, but perception decides what is visible in the current cycle. section 2.2.1 section 3.1.3
The context window is a limited treatment budget, not a database. More information can improve a decision until it dilutes attention or displaces more useful evidence. Capacity, attention quality, and token cost all constrain selection; the useful fill level depends on the task and model. section 3.1 section 3.1.2
Table 3.1 The perception engineer's trilemma
|
Dimension
|
Constraint
|
Consequence
|
|
Token budget
|
Context windows have finite, model- and configuration-dependent capacity. A 50-cycle task with one 2K-token tool result per cycle consumes 100K tokens before counting anything else.
|
You cannot include everything.
|
|
Attention quality
|
Even within the budget, attention degrades with length. Hong et al. (2025) tested 18 LLMs and found "increasingly unreliable performance as input length grows" [3].
|
More context often means worse reasoning.
|
|
Token cost
|
Uncached input is processed again on each call, while eligible prefixes may receive a lower cached rate. Input-heavy agent workloads therefore make context reuse economically significant [4].
|
Wasteful perception drains your budget.
|
The three constraints explain why fitting information into the window is not enough to make it useful. section 3.1.2
A perception pipeline can combine patterns as the task requires:
- Context Triage assigns priority tiers and chooses what to load now or defer. Priority expresses intent; it does not by itself enforce rules. section 3.8 section 3.1.3
- Semantic Compaction clears or summarizes accumulated context, while retaining decision-critical evidence until the decision is resolved. section 3.8
- Progressive Discovery gathers context in a forage-focus-deepen cycle, trading exploration cost among task-time search, session setup, and index maintenance. section 3.8
- Multi-Modal Fusion selects a representation suited to the decision: preserve images when spatial information matters and use compact text for suitable symbolic or nonspatial evidence. section 3.8
A context-aware code reviewer
A diff-only reviewer can be extended to discover modified files, their Python imports, relevant tests, and project configuration. It sorts discoveries by priority and greedily fills a token budget, recording selections and drops. Modified files rank highest, but the algorithm can still skip one if it does not fit; a priority label is not an inclusion guarantee. section 3.7
Listing 3.8 Argus triage: fit discoveries into the token budget (ch03-perception/argus/perception.py)
# Triage: sort by priority, fill greedily
priority_order = {"modified": 0, "imported": 1, "test": 1, "config": 2}
contexts.sort(key=lambda c: priority_order.get(c.relevance, 3))
trace.files_discovered = len(contexts)
trace.tokens_considered = sum(c.tokens for c in contexts)
selected, used = [], 0
for ctx in contexts:
if used + ctx.tokens <= budget: #A
selected.append(ctx)
used += ctx.tokens
else:
trace.dropped_files.append((ctx.path, ctx.relevance))
trace.files_selected = len(selected)
trace.files_dropped = trace.files_discovered - trace.files_selected
trace.tokens_selected = used
logger.info(
"Perception: %d/%d files selected (%.0f%% selectivity), "
"%d tokens used of %d budget",
trace.files_selected, trace.files_discovered,
trace.selectivity * 100, used, budget,
)
if trace.dropped_files: #B
logger.warning("Dropped: %s", trace.dropped_files)
return selected, trace
The triage implementation makes the budget limit and dropped-file trace explicit. section 3.7
Evaluate perception against the workload rather than context size alone. Track repeated retrieval, token spend per successful outcome, and reliable use of critical evidence; establish model and workload baselines before setting thresholds. Compression can save space but can also erase evidence needed to avoid repeating a failure. section 3.8
