liveBook · Manning

Chapter 3: Perception: What your agent sees determines what it does

What you will learn in chapter 3 of Designing AI Agents

The chapter introduction shows why an agent can fail despite having the right evidence: perception depends on how useful information is selected and presented, not merely on how much context is supplied.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduction shows why an agent can fail despite having the right evidence: perception depends on how useful information is selected and presented, not merely on how much context is supplied.

Key terms: perception, context window, useful context, perception engineering, bottleneck

3.1 The context window as a treatment budget

This section explains why a context window is not uniform like RAM and why perception must decide what enters its limited budget.

Key terms: context window, RAM, perception, context budget, token

3.1.1 How attention varies by position

This section explains why information in the middle of a long agent context is less reliably used than information near its beginning or end.

Key terms: U-shaped performance curve, dead zone, context, instruction adherence, remaining-capacity estimation, context bloat

3.1.2 The three-way squeeze

This section explains why agent perception must select coherent, decision-relevant context rather than simply fill the context window.

Key terms: perception, high-signal tokens, coherence, context window, context fill level, reasoning quality, iatrogenic context, attention, token leverage, uncached call

3.1.3 How production agents solve this

Claude Code manages perception by prioritizing session context, loading Skills on demand, isolating sub-agent work, and routing tool output according to its usefulness.

Key terms: context window, CLAUDE.md, instruction hierarchy, auto memory, Perceive-Reason-Act (PRA) cycle, Agent Skills, progressive disclosure, SKILL.md, Knowledge ROI, sub-agents

3.1.4 Measuring perception quality

This section shows how to measure whether an agent’s perception pipeline preserves useful information while controlling cost.

Key terms: re-read ratio, triage, compaction, token spend per successful outcome, input tokens, evaluation set, attention-zone coverage, task-specific position tests, workload baseline, perception policy

3.2 Pattern: Context triage

Introduces Context Triage as a way to give each agent judgment the most relevant evidence instead of filling its context with everything available.

Key terms: evidence boundary, perception stages, Context Triage, Perception × Route, priority tiers, context window, priority policy, useful context budget, runtime, task success

3.2.1 The emergency room

The emergency-room analogy explains why an agent must prioritize which evidence enters its limited context window rather than admit inputs indiscriminately.

Key terms: triage, severity tier, Context Triage, context window, perception policy, class of evidence

3.2.2 The four tiers

This section explains how Context Triage allocates limited context across four priority tiers and how to decide what can safely be deferred.

Key terms: Context Triage, P0 (Critical), P1 (Important), P2 (Supporting), P3 (Deferrable), lightweight handles, P3, recoverability, business consequences, policy-protected P0 or P1 evidence

3.2.3 Context triage in Claude Code

The section shows how configured priorities and protected evidence make context triage concrete in production agents.

Key terms: CLAUDE.md hierarchy, Enterprise level, User level, P0, P2, P1, protected class, release condition, failed actions, observations

3.2.4 Building it

This section explains how context triage orders items for the context window and keeps deferrable items available without loading them.

Key terms: priority classification, sort key, error traces, recency, relevance scoring, deferrable items, handles list, just-in-time retrieval

3.2.5 When it breaks

Explains two ways context triage can fail and how to recognize or mitigate them.

Key terms: priority classification, deferrable, P3, PRA cycle, promote, context budget, aggressive triage, dependencies, false positives, re-reads

3.3 Pattern: Semantic compaction

This section introduces Semantic Compaction as a way to keep long agent sessions usable without losing evidence needed for the next decision.

Key terms: Semantic Compaction, Context Triage, context window, decision-critical evidence, verbatim, compaction boundary

3.3.1 The moment the window fills

This section explains why compacting an agent’s context must preserve decision-critical evidence and provide a way to reload original material.

Key terms: context compaction, active context, decision-critical evidence, traces, lossy, pointers, original material, current state, source evidence, constraints

3.3.2 Three levels of compression

The section explains three increasingly aggressive ways to compact an agent’s context while retaining information needed for its current work.

Key terms: compaction, tool result clearing, conversation summarization, structured recap, recent turns, progressive summarization, fidelity, compression, drift

3.3.3 Protect decision-critical evidence

This section explains how to keep decision-critical evidence intact during compaction so an agent can learn from the results of its actions.

Key terms: compaction policy, protected class, decision-critical evidence, full fidelity, protection predicate, retention policy, failed action, resulting observation

3.3.4 Building it

Explains how to compact an agent’s context while limiting information loss and preserving what matters.

Key terms: two-level strategy, tool results, summary prompt, information loss

3.3.5 When it breaks

This section shows how to detect and mitigate semantic compaction failures, especially when compaction interrupts unfinished reasoning, and clarifies what compaction manages.

Key terms: Compaction failure, summary-to-source retrieval, uncompacted evaluation baseline, compaction boundary, active reasoning chain, completed conversation segments, decision point, Semantic Compaction, Context Triage, current session

3.4 Pattern: Progressive discovery

Introduces Progressive Discovery as a way to gather task-relevant evidence without loading an entire environment, and frames the choice of when to pay for discovery.

Key terms: Progressive Discovery, task boundary, evidence-bearing links, working set, per-task search, session setup, persistent index, freshness obligation, missed dependencies

3.4.1 From an entry point to working context

Shows how progressive discovery builds working context from a task-specific entry point, seeking enough evidence for the next move rather than loading everything.

Key terms: Progressive Discovery, working context, bounded rationality, Context Triage, Semantic Compaction

3.4.2 Forage, then focus, then deepen

Explains how progressive discovery narrows an agent’s search from broad, cheap scans to targeted, deeper inspection while keeping token costs low.

Key terms: progressive discovery, information foraging theory, patches, token cost per operation, token leverage, glob, grep

3.4.3 The production spectrum

This section compares how production agents discover codebase context and explains when on-demand exploration, indexing, or task-specific context is useful.

Key terms: on-demand foraging, Plan mode, Act mode, session-level indexes, vector embeddings, persistent organizational indexes, Context Engine, persistent index, disposable task contexts, freshness

3.4.4 Building it

3.4.5 The progressive disclosure connection

The section shows how Claude Code’s Agent Skills apply progressive discovery by loading skill information in stages.

Key terms: Agent Skills, progressive discovery, forage-focus-deepen, YAML header, markdown body, SKILL.md

3.4.6 When it breaks

Explains where progressive discovery breaks down and how to respond when file names or project structure do not lead the agent to useful evidence.

Key terms: progressive discovery, forage phase, glob, grep-heavy exploration, CLAUDE.md, README.md, forage step, text, Multi-Modal Fusion

3.5 Pattern: Multi-modal fusion

Multi-Modal Fusion chooses how to represent each input so the agent retains the evidence needed for its next decision.

Key terms: Multi-Modal Fusion, representation, Unreal Blueprint, build log, conversion, original, outcome accuracy, latency, token cost

3.5.1 When text is not enough

This subsection explains why the form of an agent’s input matters to what it can reason about.

Key terms: representation, input form, relationships, reasoning

3.5.2 The representation decision

This section explains how to choose a compact representation of an input without losing the evidence needed for an agent’s decision.

Key terms: spatial relationships, vision input, compact text, token efficiency, decision evidence

3.5.3 Building it

This subsection makes multi-modal representation choices auditable and identifies how to replace a prototype spatial-information test in production.

Key terms: representation decision, spatial-information test, keyword heuristic, image description

3.5.4 When it breaks

This section explains how multi-modal fusion can fail through lost decision-relevant relationships or inaccurate image-token budgets, and how to check for both.

Key terms: multi-modal fusion, source, compact text summary, downstream error cost, image-token costs, routing policy, representation policy, downstream task

3.6 Putting it all together: The perception pipeline

This section combines four perception patterns into an adaptable pipeline that turns raw information into a selected, token-efficient context window at the start of a PRA cycle.

Key terms: perception pipeline, progressive discovery, multi-modal fusion, context triage, semantic compaction, PRA cycle, acquisition, management

3.6.1 Claude Code's perception cycle in action

A four-cycle login-test example shows how Claude Code changes what it keeps in context as it finds, fixes, and verifies a bug.

Key terms: progressive discovery, context triage, semantic compaction, P0, P1, P2, signal density, useful context

3.7 Upgrading Argus: From blind reviewer to context-aware analyst

This section upgrades Argus from reviewing a diff alone to selecting and tracking task-relevant context within a token budget.

Key terms: Progressive Discovery, modified file paths, Python imports, test files, project configuration, Context Triage, P0, token budget, priority, high-attention zone

3.8 Summary

3.9 References

Lists the sources cited in Chapter 3 on agent perception.

Where this is used later

Concepts explained in this chapter

← Chapter 2: Turn a task into an agent architecture Chapter 4: Memory: What your agent remembers shapes what it becomes →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.