liveBook · Manning

Perception

As explained in Designing AI Agents

An agent capability for identifying relevant information from inputs and loading context for subsequent processing.

Perception selects, prioritizes, and assembles information for an agent’s next reasoning step. It is the entry point to the agent’s view of the world: evidence filtered out or buried in noise may never influence the resulting plan or action. Memory can supply prior context, but perception decides what is visible in the current cycle. section 2.2.1 section 3.1.3

The context window is a limited treatment budget, not a database. More information can improve a decision until it dilutes attention or displaces more useful evidence. Capacity, attention quality, and token cost all constrain selection; the useful fill level depends on the task and model. section 3.1 section 3.1.2

Table 3.1 The perception engineer's trilemma
Dimension
Constraint
Consequence
Token budget
Context windows have finite, model- and configuration-dependent capacity. A 50-cycle task with one 2K-token tool result per cycle consumes 100K tokens before counting anything else.
You cannot include everything.
Attention quality
Even within the budget, attention degrades with length. Hong et al. (2025) tested 18 LLMs and found "increasingly unreliable performance as input length grows" [3].
More context often means worse reasoning.
Token cost
Uncached input is processed again on each call, while eligible prefixes may receive a lower cached rate. Input-heavy agent workloads therefore make context reuse economically significant [4].
Wasteful perception drains your budget.

The three constraints explain why fitting information into the window is not enough to make it useful. section 3.1.2

A perception pipeline can combine patterns as the task requires:

  • Context Triage assigns priority tiers and chooses what to load now or defer. Priority expresses intent; it does not by itself enforce rules. section 3.8 section 3.1.3
  • Semantic Compaction clears or summarizes accumulated context, while retaining decision-critical evidence until the decision is resolved. section 3.8
  • Progressive Discovery gathers context in a forage-focus-deepen cycle, trading exploration cost among task-time search, session setup, and index maintenance. section 3.8
  • Multi-Modal Fusion selects a representation suited to the decision: preserve images when spatial information matters and use compact text for suitable symbolic or nonspatial evidence. section 3.8

A context-aware code reviewer

A diff-only reviewer can be extended to discover modified files, their Python imports, relevant tests, and project configuration. It sorts discoveries by priority and greedily fills a token budget, recording selections and drops. Modified files rank highest, but the algorithm can still skip one if it does not fit; a priority label is not an inclusion guarantee. section 3.7

Listing 3.8 Argus triage: fit discoveries into the token budget (ch03-perception/argus/perception.py)
    # Triage: sort by priority, fill greedily
    priority_order = {"modified": 0, "imported": 1, "test": 1, "config": 2}
    contexts.sort(key=lambda c: priority_order.get(c.relevance, 3))
 
    trace.files_discovered = len(contexts)
    trace.tokens_considered = sum(c.tokens for c in contexts)
 
    selected, used = [], 0
    for ctx in contexts:
        if used + ctx.tokens <= budget:  #A
            selected.append(ctx)
            used += ctx.tokens
        else:
            trace.dropped_files.append((ctx.path, ctx.relevance))
 
    trace.files_selected = len(selected)
    trace.files_dropped = trace.files_discovered - trace.files_selected
    trace.tokens_selected = used
    logger.info(
        "Perception: %d/%d files selected (%.0f%% selectivity), "
        "%d tokens used of %d budget",
        trace.files_selected, trace.files_discovered,
        trace.selectivity * 100, used, budget,
    )
    if trace.dropped_files:  #B
        logger.warning("Dropped: %s", trace.dropped_files)
 
    return selected, trace

The triage implementation makes the budget limit and dropped-file trace explicit. section 3.7

Evaluate perception against the workload rather than context size alone. Track repeated retrieval, token spend per successful outcome, and reliable use of critical evidence; establish model and workload baselines before setting thresholds. Compression can save space but can also erase evidence needed to avoid repeating a failure. section 3.8

How Designing AI Agents covers it

How deep it goes
Develops prioritization, compaction, discovery, multimodal representation, and observability through a budgeted implementation and operational trade-offs.
Examples
Contrasts a noisy pull-request review with a code reviewer that discovers imports, tests, and configuration before triaging files.
What you can do after
Build a budgeted perception pipeline that discovers, prioritizes, and traces review context.
How it connects
Connects context engineering with memory, reasoning, action, and the context window.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Agents. The section links point to the parts of the book it draws on.