Chapter 11: Patterns in the wild: Three end-to-end case studies
The introduction illustrates evidence-driven architecture changes and the need for exact rules at agent system boundaries.
After this chapter you will be able to
- Judgment Versus Structural Checks. Explain why the modeling prototype replaced repeated self-review with structural checks and a bounded repair attempt while retaining model judgment for names and modeling detail.
- Exact Boundaries and Evidence. Explain how agent route choice relates to exact boundary rules and evidence-based architecture decisions.
- Context-bound case interpretation. Explain what to ask about a case’s evidence, supported claim, operating bounds, and warranted architecture action before applying it.
- Evidence and architecture decisions. Explain how an evidence class limits a claim without determining the size of an architecture change, and contrast what fixtures and field accounts can reveal.
- Constructed and executable cases. Distinguish an illustrative design exercise from a repository-executed case by what each can establish and where its claims stop.
- Recorded benchmark claim limits. Describe what a recorded benchmark supports and why readers cannot independently rerun it from the published package.
- Practitioner and production accounts. Distinguish the claims supported by an attributed practitioner account and a public production report from the conclusions those accounts cannot establish.
- Keep Repair Recompose Decisions. Distinguish when evidence supports keeping, repairing, or recomposing an agent architecture.
- Simplification And Retirement. Distinguish simplifying an architecture from retiring an obsolete decision or component.
- Zero-pattern baseline decision. Explain why the support workflow keeps its deterministic route rather than adding an agent pattern.
- Salary template execution task. Describe the onboarding problem and the required salary-template execution task.
- Salary group completion boundary. Describe where the salary-group task begins and what must happen for it to finish.
- Business identifier ownership. Explain why business identifiers and narrative meaning need different owners in this task.
- Tool Using ReAct Baseline. Describe how Dongfang Yiteng's initial design used its domain APIs, a Skill, and a ReAct loop, and what the baseline tested.
- Unenforced Task Dependencies. Explain why natural-language ordering instructions were insufficient as the task reached known dependencies.
- Dependency Drift and Graph Control. Explain the observed sequence failures and how Plan-and-Execute managed task dependencies after exploration.
- Runtime Planning Versus Workflow. Distinguish the linear salary path from the broader reason for using Plan-and-Execute.
- Ordering and Identifier Provenance. Explain what the secured task order still could not establish about the import step.
- Identifier transfer and validity. Trace how the matching service's identifier reached the import call, and explain why schema validity did not guarantee the correct business identity.
- Business identity source of truth. Explain why retries or prompt changes did not resolve the ownership and origin problem, and distinguish task dependencies from business-value origin.
- Identifier Handoff Diagnosis. Explain how the run trace localized the loss of the required identifier to the next model call.
- Workspace task progress. Explain how Workspace tracks task progress and determines when a downstream task is ready.
- Exact business value preservation. Explain how SessionState carries exact tool-produced values into later calls without requiring the model to regenerate them.
- Narrative goal and meaning. Describe the Anchor, Ledger, and Collection roles in SessionNarrative and distinguish narrative context from exact business values.
- Request routing and template provenance. Trace how the request is interpreted, routed, and matched to a template while preserving the template's origin.
- Snapshot and approval gates. Explain how the replay protects the current configuration and obtains a valid decision before import.
- Execution verification and recovery. Trace how the import uses preserved identifiers, how completion is verified, and how recovery retains state lineage.
- Two-stage recomposition decision. Explain why the team retained the planned task graph but added SessionState to repair a separate failure.
- Narrower architecture alternatives. Distinguish a predefined route from the reported planner and explain which task conditions favor the alternatives.
- Evidence limits and reopening. Describe what the bounded replay establishes and when the architecture decision should be reopened.
- Model Repository and Diagrams. Explain how the modeling tools relate diagram elements to repository records and why those records matter for later work.
- Prototype Design Changes. Describe the two design changes reported during local experimentation with AI4MBSE.
- Model and diagram consistency. Explain what must be true beyond producing a convincing diagram when adding a controller and connecting it to an existing sensor.
- Bounded modeling operation. Explain how limiting an operation to one diagram type and one resolved project location supports checking and undoing changes, and what happens when the location is ambiguous.
- Progress tracking across clarification. Describe what the saved operation retains while awaiting clarification and why the runtime resumes it rather than recognizing intent again from the short answer.
- Dependency ordered modeling chain. Trace the sequence from accepted elements to relationships to the diagram view, and explain how checked results constrain each stage.
- Failure isolation and runtime ownership. Explain why splitting the response localizes failures and how one runtime can run the known sequence without separate agents or a planner.
- Stage specific context assembly. Describe the context needed by the element, relationship, and view stages, and explain why loading every project detail and diagram rule into every call is undesirable.
- Review Loop Removal. Explain why the prototype removed its post-write self-review cycle.
- Computable Checks and Judgment. Distinguish structural conditions the runtime can check from modeling qualities that still need model or human judgment.
- Bounded Repair and Blocking. Describe when a failed check can receive a bounded repair and when the operation must wait for user input or remain blocked.
- Observable write outcomes. Describe what the write result records and how the prototype distinguishes outcomes, including empty changes.
- Model and view verification. Explain why the resulting model and displayed view both need inspection after a write.
- Comparative boundary testing. Describe how to compare the design against another version and test its boundary responsibilities in a new application.
- Division of runtime responsibilities. Explain how the reusable design separates semantic proposals, acceptance checks, and the final write, and describe the roles of its three patterns.
- Transfer and completion boundaries. Explain how to assess whether another application needs a similar design, using its completion boundary and obligations.
- Changing Conditions and Validity. Explain why an agent design that passed its original evaluation can later produce an incorrect analysis.
- Analytics context and interpretation. Explain what context an agent must establish to investigate a fall in weekly active users, and why correct SQL alone may not explain the change.
- State authority and evidence. Describe the state, authority, and evidence boundaries in the analytics task.
- Tool capability surface. Explain why the team restricted and consolidated the data agent's tool calls.
- Goal-level control guidance. Explain why higher-level goal guidance replaced highly prescriptive route instructions.
- Preconditions versus search strategy. Distinguish an enforced consequential-action precondition from an investigation strategy that can change with evidence.
- Golden SQL Evaluation. Describe how the team evaluates generated SQL and maintains the golden set as workloads change.
- Localized Evidence for Decisions. Explain how an evaluation failure and execution artifacts support an architecture action without choosing a pattern automatically.
- Tool and guidance decisions. Explain why the case simplifies overlapping tools and repairs overly prescriptive guidance.
- Evaluation driven decision reopening. Explain how evaluations and a decision ledger support reopening an architectural decision without predetermining the next fix.
- Diagnosed deficits and choices. Explain how a diagnosed deficit directs the choice of architecture or check, using the execution-chain and modeling cases.
- Preserved intermediate artifacts. Explain why an intermediate artifact should be preserved for the next component, including a human reviewer.
- Dependencies and parallel work. Distinguish work that can plausibly run in parallel from stages whose dependencies require coordination.
- Versioned architecture decisions. Explain why architecture decisions and the reasons for replacing them should be recorded over time.
- Evidence for removing complexity. Explain when evidence justifies removing an architectural component and why the result should be retained.
- Trace-Based Failure Localization. Explain how to use a failure trace to decide where a composition needs to change.
- Problem and contract mismatch. Explain how a problem or contract mismatch can make an apparently successful agent fail the user’s task.
- Contract reassessment and retirement. Describe what to do when the task or authority is wrong, and when to retire the architecture decision.
- Task Pattern Fit. Explain how a pattern that is valid in isolation can fail to fit a task, using an example from the section.
- Replacement Composition Evaluation. Describe how to respond when evidence shows a pattern's preconditions no longer fit, including how to evaluate a replacement.
- Pattern Seam Failures. Explain what a seam failure is and identify how it can appear at a handoff.
- Version Bound Approval. Explain how the salary replay binds a receipt and approval to the correct template and what happens if the template changes.
- Uncertain Import Recovery. Describe how to handle a timed-out import that may already have written a salary group.
- Repair Versus Recomposition. Distinguish when to repair a seam from when to recompose the adjacent components.
- Parameter and model diagnosis. Explain how parameter or model limitations can cause failure even when the architecture and evidence path are appropriate, and describe how to compare alternatives.
- Accepted card tuning boundaries. State when tuning can remain within the accepted card and when a superseding card is required.
- Workload Shift Failure Signals. Explain how changes in workload or scale can expose failures that an evaluation set or overall average conceals.
- Workload Reassessment and Selection. Describe how to reassess pattern choices when the workload changes, including the limits of earlier acceptance evidence.
- Evidence Across Task Boundaries. Trace the evidence needed from a local mechanism through handoffs, completed task, and operating conditions in the salary-import example.
- Risk Based Confidence Policy. Explain how risk and uncertainty shape the amount and kind of architecture testing.
- Representative Stratified Workloads. Describe how to construct and report a representative, stratified workload for architecture comparison.
- Controlled Multimetric Comparison. Explain how a fixed baseline and multiple metrics support a fair architecture comparison.
- Architecture decision ledger purpose. Explain how the architecture decision ledger connects design rationale to the evidence behind acceptance and later changes.
- Versioned evidence and reopening. Describe how to preserve and update the decision record when evidence or assumptions change, including where reconsideration resumes.
- Evidence-driven pattern application. Describe how to use a verifiable task, a simple baseline, failure evidence, and a smaller test to decide which patterns the system needs.
- External evidence and evaluations. Explain how to use published engineering evidence and preserved evaluations when models and tools change.
What it builds on
- Step 2: Run the smallest end-to-end baseline (chapter 10) · Running the smallest end-to-end baseline
- Step 3: Diagnose the binding deficit (chapter 10) · Diagnosing a binding deficit before selecting patterns
- The Task Completion Contract (chapter 10) · Bounding a task with a completion contract.
- Pattern: Plan-and-Execute (chapter 6) · Plan-and-Execute as an action pattern.
- What to observe: Metrics locate the problem; trajectories explain the path (chapter 9) · The role of trajectories in explaining an agent's path
- Draw the task spine and four continuity lines (chapter 10) · The task spine and four continuity lines used to model a task.
- Pattern: Progress tracking (chapter 4) · Progress tracking as a way to preserve work across interruptions
- Pattern: Prompt Chaining (chapter 6) · Prompt Chaining and the role of artifacts passed between stages
Section by section
Introduction
The introduction illustrates evidence-driven architecture changes and the need for exact rules at agent system boundaries.
- Judgment Versus Structural Checks: Explain why the modeling prototype replaced repeated self-review with structural checks and a bounded repair attempt while retaining model judgment for names and modeling detail.
Why it matters: This distinction helps you reserve model judgment for decisions that actually need it. - Exact Boundaries and Evidence: Explain how agent route choice relates to exact boundary rules and evidence-based architecture decisions.
Why it matters: Autonomous choices need clear boundaries and evidence to support the architecture around them.
Key terms: self-review loop, structural checks, repair attempt, authority, shared state, external effects, claims of completion, evidence
11.1 How to read an architecture case
This section frames how to interpret architecture cases without mistaking context-bound evidence for a universal rule.
- Context-bound case interpretation: Explain what to ask about a case’s evidence, supported claim, operating bounds, and warranted architecture action before applying it.
Why it matters: These questions help you use a case without turning its particular context into a universal design rule.
Key terms: evidence class, workload, operating conditions, architecture action
11.1.1 Five evidence classes
This section distinguishes five evidence classes by the claims they support and the limits they place on architectural conclusions.
- Evidence and architecture decisions: Explain how an evidence class limits a claim without determining the size of an architecture change, and contrast what fixtures and field accounts can reveal.
Why it matters: Matching the claim to the evidence prevents either overstating a result or dismissing useful evidence. - Constructed and executable cases: Distinguish an illustrative design exercise from a repository-executed case by what each can establish and where its claims stop.
Why it matters: These distinctions keep a useful example or reproducible implementation from being mistaken for proof of production performance elsewhere. - Recorded benchmark claim limits: Describe what a recorded benchmark supports and why readers cannot independently rerun it from the published package.
Why it matters: A preserved result can be useful evidence without providing everything needed to rerun it independently. - Practitioner and production accounts: Distinguish the claims supported by an attributed practitioner account and a public production report from the conclusions those accounts cannot establish.
Why it matters: Real-world accounts reveal useful design and operating context, but their reported outcomes should not automatically be treated as causal or transferable proof.
Key terms: evidence classes, architecture action, deterministic fixture, field account, illustrative design exercise, repository-executed case, code, fixtures, expected outcomes, recorded benchmark
11.1.2 Five architecture actions
This section distinguishes five evidence-driven architecture actions and explains how to record changes to decisions.
- Keep Repair Recompose Decisions: Distinguish when evidence supports keeping, repairing, or recomposing an agent architecture.
Why it matters: These distinctions help you choose a response proportional to what the evidence shows. - Simplification And Retirement: Distinguish simplifying an architecture from retiring an obsolete decision or component.
Why it matters: Removing unnecessary complexity is different from ending something that is no longer needed. - Architecture Decision Ledger: Describe what to retain in an architecture decision ledger when simplifying or retiring a decision.
Why it matters: A decision history makes later changes understandable rather than leaving their rationale behind.
Key terms: Keep, Repair, Recompose, gates, operating point, consequential seam, Simplify, Retire, architecture decision ledger, accepted choice
11.1.3 A zero-pattern control: When evidence says stop
An illustrative support workflow shows when to keep a deterministic baseline without adding agent patterns and what evidence would reopen that decision.
- Zero-pattern baseline decision: Explain why the support workflow keeps its deterministic route rather than adding an agent pattern.
Why it matters: A working baseline can be the right architecture when added autonomy has no demonstrated problem to solve. - Decision reopen conditions: Describe the changes that would reopen the decision to use no agent pattern.
Why it matters: A simple architecture should be reconsidered when the workload exceeds what its policies and encoded choices can reliably handle.
Key terms: deterministic route, general agent loop, reasoning or collaboration deficit, exception volume, policy surface, cross-session causal reasoning, tool choice
11.2 Case 1: From tool calls to a state-preserving execution chain
This section introduces a salary-template execution task and distinguishes the practitioner account from the case’s teaching reconstructions.
- Salary template execution task: Describe the onboarding problem and the required salary-template execution task.
Why it matters: These requirements define what the agent must accomplish without making users master the configuration screens. - Case evidence and reconstruction: Distinguish the practitioner-supported case facts from the teaching reconstructions.
Why it matters: This distinction prevents treating every detail of the teaching example as a reported production fact.
Key terms: salary-group template, identifier, snapshot, import, attributed practitioner account, teaching reconstructions, authoritative readback
11.2.1 Bound the completed task
This section bounds the salary-group task and explains why business identifiers need different ownership from narrative meaning.
- Salary group completion boundary: Describe where the salary-group task begins and what must happen for it to finish.
Why it matters: A clear completion boundary keeps the agent's response from being mistaken for the business outcome. - Business identifier ownership: Explain why business identifiers and narrative meaning need different owners in this task.
Why it matters: Preserving business identity prevents small changes from becoming consequential errors.
Key terms: salary group, authenticated user, business system, tenant, business identifier, business identity, narrative meaning, state transitions
11.2.2 The smallest plausible baseline
The section describes a tool-using ReAct baseline and identifies its lack of runtime enforcement for known task dependencies.
- Tool Using ReAct Baseline: Describe how Dongfang Yiteng's initial design used its domain APIs, a Skill, and a ReAct loop, and what the baseline tested.
Why it matters: This baseline shows what the team could test before adding more structure. - Unenforced Task Dependencies: Explain why natural-language ordering instructions were insufficient as the task reached known dependencies.
Why it matters: Known dependencies can be described in instructions without being enforced by the system.
Key terms: domain APIs, tools, Skill, ReAct loop, tool result, dependencies, matching, import, snapshot, recovery path
11.2.3 Failure one: Fixed steps drift
This section explains how Plan-and-Execute addressed skipped steps and dependency drift while leaving identifier provenance unresolved.
- Dependency Drift and Graph Control: Explain the observed sequence failures and how Plan-and-Execute managed task dependencies after exploration.
Why it matters: Explicit task state helps prevent an agent from advancing across unmet dependencies. - Runtime Planning Versus Workflow: Distinguish the linear salary path from the broader reason for using Plan-and-Execute.
Why it matters: A sequence of steps alone does not justify adding runtime planning. - Ordering and Identifier Provenance: Explain what the secured task order still could not establish about the import step.
Why it matters: Correct step order does not by itself guarantee that the right value crosses a handoff.
Key terms: Skill, ReAct, Plan-and-Execute, orchestrator, task DAG, scheduler, predefined workflow, tasks, dependencies, task graph
11.2.4 Failure two: A valid argument loses its origin
This section explains how a valid import call can lose the origin of a business identifier and why conversation history is insufficient as its source of truth.
- Identifier transfer and validity: Trace how the matching service's identifier reached the import call, and explain why schema validity did not guarantee the correct business identity.
Why it matters: A structurally valid call can still act on the wrong business entity. - Business identity source of truth: Explain why retries or prompt changes did not resolve the ownership and origin problem, and distinguish task dependencies from business-value origin.
Why it matters: Reliable business identity requires more than improving the prompt or knowing which task depends on which.
Key terms: template_id, JSON Schema, business identifier, task graph, node dependencies, source of truth, business identity
11.2.5 The trace localizes the handoff
The run trace localizes the loss of a required identifier to the handoff between a matching tool and the next model call.
- Identifier Handoff Diagnosis: Explain how the run trace localized the loss of the required identifier to the next model call.
Why it matters: The trace distinguishes a tool that produced the required identifier from a later handoff where its guarantee was lost.
Key terms: trace, matching tool, identifier, model call
11.2.6 Three runtime structures for three kinds of truth
This section explains how three runtime structures separately preserve task progress, exact business values, and the narrative context of a run.
- Workspace task progress: Explain how Workspace tracks task progress and determines when a downstream task is ready.
Why it matters: A summary of progress is not a reliable substitute for execution state. - Exact business value preservation: Explain how SessionState carries exact tool-produced values into later calls without requiring the model to regenerate them.
Why it matters: Preserving the original values prevents a model-generated identifier from replacing the one produced by a tool. - Import contract and approval: Describe how the illustrative import contract and approval receipt constrain a call when required information or reviewed state is missing or changed.
Why it matters: The call is blocked rather than guessing when its required or approved context no longer holds. - Narrative goal and meaning: Describe the Anchor, Ledger, and Collection roles in SessionNarrative and distinguish narrative context from exact business values.
Why it matters: The model needs a useful account of the task without turning that account into the source of exact identifiers.
Key terms: Workspace, task graph, dependencies, SessionState, state coordinates, tool receipt, template_id, snapshot_id, contract, preconditions
11.2.7 Replay the task with the new ownership
Replays a salary-group request to show how runtime state, approval, execution, and verification preserve continuity across agent handoffs.
- Request routing and template provenance: Trace how the request is interpreted, routed, and matched to a template while preserving the template's origin.
Why it matters: This handoff keeps the selected template tied to the request and the tool that produced it. - Snapshot and approval gates: Explain how the replay protects the current configuration and obtains a valid decision before import.
Why it matters: The gates connect a consequential import to both a recovery basis and a current authorized decision. - Execution verification and recovery: Trace how the import uses preserved identifiers, how completion is verified, and how recovery retains state lineage.
Why it matters: A successful tool call alone does not establish that the intended business result was delivered or can be recovered. - Four continuity lines: Describe the four continuity lines exposed by the replay and distinguish them from new components.
Why it matters: The continuity lines provide a way to check what must remain connected across the whole task.
Key terms: ReAct, Tool Dispatch, SessionNarrative, SessionState, template_id, snapshot, Progress Tracking, approval receipt, snapshot_id, import_receipt
11.2.8 Evidence, decision, and reopen conditions
This section evaluates a two-stage architectural recomposition, compares narrower alternatives, and sets limits and reopening conditions for the decision.
- Two-stage recomposition decision: Explain why the team retained the planned task graph but added SessionState to repair a separate failure.
Why it matters: Fixing one failure does not prove that a different handoff is correct. - Narrower architecture alternatives: Distinguish a predefined route from the reported planner and explain which task conditions favor the alternatives.
Why it matters: The architecture should match the task's variability and the actual failure rather than add patterns indiscriminately. - Evidence limits and reopening: Describe what the bounded replay establishes and when the architecture decision should be reopened.
Why it matters: A defensible architecture decision states both what its evidence shows and what could change the decision.
Key terms: planned task graph, SessionState, parameter binding, Prompt Chaining, Plan-and-Execute, ReAct, Fan-Out/Gather, predefined route, bounded replay, state contracts
11.3 Case 2: Deciding what a modeling agent should judge
This introduction presents AI4MBSE's modeling context and the design changes explored in its research prototype.
- Model Repository and Diagrams: Explain how the modeling tools relate diagram elements to repository records and why those records matter for later work.
Why it matters: A diagram change can affect records that subsequent engineering work relies on. - Prototype Design Changes: Describe the two design changes reported during local experimentation with AI4MBSE.
Why it matters: These changes frame the case's decision about where generated judgment is needed and where checks can be programmed.
Key terms: model repository, typed elements, relationships, identifier, checked plans, model self-review, structural checks
11.3.1 Start with the model behind the picture
The section defines a modeling task’s correct result, bounds its changes, and preserves unfinished work while awaiting location clarification.
- Model and diagram consistency: Explain what must be true beyond producing a convincing diagram when adding a controller and connecting it to an existing sensor.
Why it matters: A plausible picture can hide a model that does not satisfy the requested change. - Bounded modeling operation: Explain how limiting an operation to one diagram type and one resolved project location supports checking and undoing changes, and what happens when the location is ambiguous.
Why it matters: A clear operation boundary makes changes checkable without guessing where they belong. - Progress tracking across clarification: Describe what the saved operation retains while awaiting clarification and why the runtime resumes it rather than recognizing intent again from the short answer.
Why it matters: Preserving unfinished work lets a brief clarification complete the original request rather than replace it.
Key terms: controller, existing sensor, diagram, diagram type, resolved project location, clarification, Progress Tracking, saved operation, current stage
11.3.2 Separate the work at its dependencies
This section separates model and diagram generation into dependency-ordered stages that pass checked artifacts and receive stage-specific context.
- Dependency ordered modeling chain: Trace the sequence from accepted elements to relationships to the diagram view, and explain how checked results constrain each stage.
Why it matters: The sequence preserves accepted structure while each later stage performs its own task. - Failure isolation and runtime ownership: Explain why splitting the response localizes failures and how one runtime can run the known sequence without separate agents or a planner.
Why it matters: Isolating failures helps preserve correct work without adding unnecessary orchestration. - Stage specific context assembly: Describe the context needed by the element, relationship, and view stages, and explain why loading every project detail and diagram rule into every call is undesirable.
Why it matters: Matching context to each stage reinforces the division of responsibility.
Key terms: Prompt Chaining, elements, relationships, view, checked artifacts, runtime, shared state, planner, independent agents, context assembly
11.3.3 Remove a review loop when the check can be computed
This section replaces an unproductive model self-review cycle with computable checks, bounded repair, and review reserved for qualitative judgments.
- Review Loop Removal: Explain why the prototype removed its post-write self-review cycle.
Why it matters: A review loop should earn its cost by improving the result it is meant to protect. - Computable Checks and Judgment: Distinguish structural conditions the runtime can check from modeling qualities that still need model or human judgment.
Why it matters: Separating mechanical checks from qualitative judgment puts review effort where another opinion can add value. - Bounded Repair and Blocking: Describe when a failed check can receive a bounded repair and when the operation must wait for user input or remain blocked.
Why it matters: A bounded repair prevents the agent from guessing its way past an unresolved condition.
Key terms: post-write self-review cycle, usable writes, bounded repair, accepted element set, dangling relationship, naming quality, modeling granularity, type mapping, project location
11.3.4 Make the write and its result observable
This section explains how to make model writes observable and evaluate their correctness across the repository, view, and application boundaries.
- Observable write outcomes: Describe what the write result records and how the prototype distinguishes outcomes, including empty changes.
Why it matters: A visible outcome helps distinguish a completed change from a blocked or empty operation. - Model and view verification: Explain why the resulting model and displayed view both need inspection after a write.
Why it matters: Checking only one representation can miss an incomplete or incorrect result. - Comparative boundary testing: Describe how to compare the design against another version and test its boundary responsibilities in a new application.
Why it matters: A fair local comparison can expose correctness losses that faster writes alone would hide. - Type translation traceability: Explain why the adapter needs an explicit type mapping and why the trace retains original and normalized names.
Why it matters: An explicit, traceable mapping makes type mismatches easier to detect and diagnose.
Key terms: write path, undo handle, no-op, model repository, canvas, completion contract, acceptance conditions, boundary, semantic check, type translation
11.3.5 Reuse the division of responsibility
This section explains how to reuse and revisit the modeling case’s division of responsibility in other applications.
- Division of runtime responsibilities: Explain how the reusable design separates semantic proposals, acceptance checks, and the final write, and describe the roles of its three patterns.
Why it matters: Separating these responsibilities helps make the proposed change and its result subject to checks. - Transfer and completion boundaries: Explain how to assess whether another application needs a similar design, using its completion boundary and obligations.
Why it matters: The right design depends on what the application must actually complete, not just on superficial similarity to the case. - Decision ledger and reopening: Describe what the decision ledger records and when the design choice should be reopened.
Why it matters: Recording changes and their reopen conditions keeps the architecture responsive to changed responsibilities.
Key terms: semantic proposals, acceptance checks, Prompt Chaining, Progress Tracking, Guardrail Sandwich, completion boundary, stored state, decision ledger, accepted scope, custom runtime
11.4 Case 3: Operating evidence reopens the architecture
Introduces a deployed data agent and explains why changing operating conditions can reopen an accepted architecture.
- Changing Conditions and Validity: Explain why an agent design that passed its original evaluation can later produce an incorrect analysis.
Why it matters: An initial evaluation cannot establish that a deployed design will remain reliable as its operating environment changes. - Data Agent Task Boundaries: Describe the data agent's path from a natural-language question to a result a person can inspect.
Why it matters: The agent's task spans finding the right information, executing a query within permissions, and making its analysis verifiable.
Key terms: deployed architecture, original evaluation, operating evidence, data discovery, business definitions, SQL generation, query plans, user's permissions, inspectable result
11.4.1 One question crosses several boundaries
This section uses an illustrative analytics question to show why explaining a change requires contextual definitions, authorized data, and traceable evidence, not just plausible SQL.
- Analytics context and interpretation: Explain what context an agent must establish to investigate a fall in weekly active users, and why correct SQL alone may not explain the change.
Why it matters: A technically plausible query can answer the wrong business question. - State authority and evidence: Describe the state, authority, and evidence boundaries in the analytics task.
Why it matters: These boundaries make the explanation traceable while keeping the analysis within the user's access rights.
Key terms: schema metadata, lineage, business definition, metric definition, state, authority, evidence, data snapshot
11.4.2 Two operating mismatches changed the design
This section explains how a data-agent team repaired overlapping tools and brittle instructions while preserving consequential boundaries, and how to compare those repairs with the earlier design.
- Tool capability surface: Explain why the team restricted and consolidated the data agent's tool calls.
Why it matters: A capability surface that makes sense to a person can still make tool selection harder for an agent. - Goal-level control guidance: Explain why higher-level goal guidance replaced highly prescriptive route instructions.
Why it matters: Guidance can preserve the intended outcome without forcing a brittle route through every question. - Preconditions versus search strategy: Distinguish an enforced consequential-action precondition from an investigation strategy that can change with evidence.
Why it matters: Flexibility in investigation should not come at the expense of safeguards on consequential actions. - Pattern handoff mapping: Describe how the section maps RAG Pipeline, Context Triage, and Tool Dispatch to the data agent's handoffs.
Why it matters: The pattern vocabulary helps locate distinct handoffs in the architecture. - Repair comparison design: Explain why the first comparison should test the reported repairs against the earlier design on the same questions rather than simply adding agents.
Why it matters: A comparison should test the changes that address the observed problems against the earlier design.
Key terms: capability surface, tool calls, higher-level goal guidance, control contract, execution path, precondition, search strategy, permissions, data provenance, evidence of completion
11.4.3 Evals make drift visible
The section explains how SQL evaluations expose drift and guide architecture changes while preserving access and answer-inspection boundaries.
- Golden SQL Evaluation: Describe how the team evaluates generated SQL and maintains the golden set as workloads change.
Why it matters: A living evaluation set makes changing query failures visible both during development and in production. - Localized Evidence for Decisions: Explain how an evaluation failure and execution artifacts support an architecture action without choosing a pattern automatically.
Why it matters: Locating the first deviation grounds an architecture change in evidence rather than a pattern guess. - Stable Trust Boundaries: Explain which access and inspection controls remain stable as the system changes its tools or guidance.
Why it matters: Stable access and inspection controls preserve the basis for user trust while the system evolves.
Key terms: golden SQL, golden set, canaries, first deviation, trace, result artifacts, architecture action, pass-through access, assumptions, execution steps
11.4.4 Decision and reopen condition
The case records tool and guidance changes while using evaluations to decide when architecture decisions need reconsideration.
- Tool and guidance decisions: Explain why the case simplifies overlapping tools and repairs overly prescriptive guidance.
Why it matters: The right change depends on whether the ambiguity comes from available capabilities or from the guidance governing their use. - Evaluation driven decision reopening: Explain how evaluations and a decision ledger support reopening an architectural decision without predetermining the next fix.
Why it matters: Recording the evidence behind a decision makes it possible to revisit the right choice when the workload reveals new drift.
Key terms: capability surface, overlapping tools, prescriptive guidance, eval-driven reopen mechanism, representative workload, failure cluster, decision ledger
11.5 Cross-case analysis
11.5.1 What the cases teach us
The section draws five evidence-driven principles for choosing, connecting, simplifying, and revisiting agent architectures.
- Diagnosed deficits and choices: Explain how a diagnosed deficit directs the choice of architecture or check, using the execution-chain and modeling cases.
Why it matters: Starting with the actual deficit helps you choose a remedy that fits the task rather than adding patterns by habit. - Preserved intermediate artifacts: Explain why an intermediate artifact should be preserved for the next component, including a human reviewer.
Why it matters: The next step can only use facts and candidates that survive the handoff. - Dependencies and parallel work: Distinguish work that can plausibly run in parallel from stages whose dependencies require coordination.
Why it matters: Parallel work is useful only when its dependencies and coordination costs fit the task. - Versioned architecture decisions: Explain why architecture decisions and the reasons for replacing them should be recorded over time.
Why it matters: A decision history helps future engineers understand when and why an architecture stopped fitting. - Evidence for removing complexity: Explain when evidence justifies removing an architectural component and why the result should be retained.
Why it matters: Evidence can justify a simpler system while preserving a basis for revisiting the choice.
Key terms: diagnosed deficit, Plan-and-Execute, Prompt Chaining, deterministic check, intermediate artifact, SessionState, accepted element set, review queue, parallel work, accepted elements
11.6 A failure taxonomy for composed agents
This section introduces diagnosing composed-agent failures by tracing contract deviations to the relevant architectural layer, illustrated by two case examples.
- Trace-Based Failure Localization: Explain how to use a failure trace to decide where a composition needs to change.
Why it matters: Starting from the first contract deviation helps you choose what to change rather than treating only the visible failure. - Contrasting Case Failure Signals: Distinguish what the two case examples reveal about where a failure lies.
Why it matters: These examples show why the apparent error and the actual architectural mismatch can be different.
Key terms: failure trace, contract, diagnostic layers, seam, reviewer, relationship endpoints
11.6.1 Layer 1: Problem and contract
Diagnoses problem-and-contract failures and describes when to reassess, revise, or retire an architecture decision.
- Problem and contract mismatch: Explain how a problem or contract mismatch can make an apparently successful agent fail the user’s task.
Why it matters: A good internal score does not establish that the system completed the task the user needs. - Contract reassessment and retirement: Describe what to do when the task or authority is wrong, and when to retire the architecture decision.
Why it matters: Reassessing the task prevents effort from being spent tuning a system that no longer serves a valid purpose.
Key terms: external completion boundary, internal evaluator, accepted scope, authority, ASSESS, Task Completion Contract, acceptance gate, model control
11.6.2 Layer 2: Task-pattern fit
This section explains how to recognize a task-pattern mismatch and evaluate a replacement composition.
- Task Pattern Fit: Explain how a pattern that is valid in isolation can fail to fit a task, using an example from the section.
Why it matters: A pattern's general validity does not mean it fits the work your agent must perform. - Replacement Composition Evaluation: Describe how to respond when evidence shows a pattern's preconditions no longer fit, including how to evaluate a replacement.
Why it matters: Checking the preconditions and comparing on the same workload helps prevent an unsupported architecture replacement.
Key terms: task-pattern fit, dependencies, required judgment, pattern's preconditions, ROUTE, SELECT, baseline, candidate, bound workload
11.6.3 Layer 3: Pattern seam
This section explains how to identify and repair pattern seam failures, using approval and recovery safeguards from the salary replay.
- Pattern Seam Failures: Explain what a seam failure is and identify how it can appear at a handoff.
Why it matters: A handoff can fail even when its components are individually useful. - Version Bound Approval: Explain how the salary replay binds a receipt and approval to the correct template and what happens if the template changes.
Why it matters: An approval must not silently authorize content that changed after it was reviewed. - Uncertain Import Recovery: Describe how to handle a timed-out import that may already have written a salary group.
Why it matters: A timeout does not prove that the external write failed, so a blind retry can duplicate its effect. - Repair Versus Recomposition: Distinguish when to repair a seam from when to recompose the adjacent components.
Why it matters: The right fix depends on whether the neighboring contracts can still coexist.
Key terms: seam failure, Task, State, Authority, Evidence, receipt, template version, digest, approval, requesting principal
11.6.4 Layer 4: Parameter and model
This section explains how to diagnose and compare parameter or model changes within a valid architecture while respecting the accepted card.
- Parameter and model diagnosis: Explain how parameter or model limitations can cause failure even when the architecture and evidence path are appropriate, and describe how to compare alternatives.
Why it matters: This helps you distinguish a tuning or model problem from a need to redesign the architecture. - Accepted card tuning boundaries: State when tuning can remain within the accepted card and when a superseding card is required.
Why it matters: This keeps tuning aligned with the accepted architectural decision and its claim.
Key terms: evidence path, loop limit, threshold, model, topology, simpler baseline, accepted card, authorized ranges, superseding card, accepted claim
11.6.5 Layer 5: Workload and scale
This section explains how workload and scale changes can invalidate earlier evaluation results and prompt a new architecture comparison.
- Workload Shift Failure Signals: Explain how changes in workload or scale can expose failures that an evaluation set or overall average conceals.
Why it matters: A good overall score can hide failures that emerge under a different workload. - Workload Reassessment and Selection: Describe how to reassess pattern choices when the workload changes, including the limits of earlier acceptance evidence.
Why it matters: Reopening the decision with targeted evidence prevents an old result from being applied beyond its demonstrated scope.
Key terms: workload, concurrency, latency tail, operating point, stratify, selection card, hierarchy, compaction, bound comparison
11.7 How much evidence is enough
This section explains how to judge whether evidence is sufficient to support an agent architecture decision at the accepted level of risk.
- Evidence Across Task Boundaries: Trace the evidence needed from a local mechanism through handoffs, completed task, and operating conditions in the salary-import example.
Why it matters: A component can pass its tests while the assembled agent still fails to complete the task safely. - Evidence Class and Coverage: Distinguish evidence class from evidence coverage.
Why it matters: Both the kind of evidence and the parts of the claim it covers matter when judging an architecture. - Risk Based Confidence Policy: Explain how risk and uncertainty shape the amount and kind of architecture testing.
Why it matters: A test threshold appropriate for a low-cost error may be inadequate for a rare, expensive failure. - Representative Stratified Workloads: Describe how to construct and report a representative, stratified workload for architecture comparison.
Why it matters: An overall score can hide the cases where an architecture needs to change. - Controlled Multimetric Comparison: Explain how a fixed baseline and multiple metrics support a fair architecture comparison.
Why it matters: A fair comparison shows whether a component helps without hiding its costs or risks. - Evidence Decision Records: Describe the respective roles of the Pattern Selection Card and Evaluation Receipt in making an evidence judgment traceable.
Why it matters: Keeping the rationale and actual evidence together lets a later team trace why the judgment changed.
Key terms: mechanism, handoff test, end-to-end evidence, operating evidence, authoritative readback, evidence class, evidence coverage, architecture claim, sample count, confidence and risk policy
11.8 The architecture decision ledger
This section explains how an architecture decision ledger preserves evidence and decision history as agent architectures change.
- Architecture decision ledger purpose: Explain how the architecture decision ledger connects design rationale to the evidence behind acceptance and later changes.
Why it matters: The ledger lets you explain not just what architecture is in use, but why it was chosen or changed. - Versioned evidence and reopening: Describe how to preserve and update the decision record when evidence or assumptions change, including where reconsideration resumes.
Why it matters: Versioned records make architectural changes traceable without erasing the evidence behind earlier decisions.
Key terms: Architecture Decision Record, architecture decision ledger, Evaluation Receipt, Pattern Selection Card, operating evidence, earliest affected phase
11.9 Taking the patterns into your own work
Apply agent patterns through small, evidence-driven tests, preserve evaluations, and share decisions with their operating conditions and limits.
- Evidence-driven pattern application: Describe how to use a verifiable task, a simple baseline, failure evidence, and a smaller test to decide which patterns the system needs.
Why it matters: This gives you a way to improve an uncertain architecture without adding parts just because they have pattern names. - External evidence and evaluations: Explain how to use published engineering evidence and preserved evaluations when models and tools change.
Why it matters: Testing external claims against your own workload helps you avoid adopting changes that do not improve your system. - Communicable decision records: Describe what to share alongside a successful result so others can question and improve the work.
Why it matters: A record of conditions and limits makes your decisions useful to other people rather than presenting success without context.
Key terms: pattern map, failed assumption, trace, operating conditions, evaluation set, workload, conditions, limits, record
11.10 Summary
11.11 References
Lists sources referenced in the chapter’s architecture case studies and analysis.
Concepts explained in this chapter
- evidence classAn evidence class is a category describing the kind of evidence available for assessing an architectural claim.
- evidence coverageThe extent to which decisive claims in a result are linked to the evidence required to support them.
- decision recordAn inspectable record of a decision or conclusion and its supporting evidence, assumptions, and rationale.
- prompt chainA workflow pattern that divides a known sequence into narrow stages, with each stage producing an artifact that passes a gate before the next stage proceeds.
- trajectoryA record of events and operations during a system run, arranged to show its progression and support diagnosis.
- progress trackingA persistence pattern that records task state and completed work so an interrupted agent can resume execution.
- evaluationEvaluation is the systematic assessment of a result against criteria for quality, correctness, or suitability.
- plan-and-executeAn agent pattern in which planning precedes execution of the planned steps.
- source of truthA source of truth is the authoritative source for a particular piece of information.
- contractAn agreement specifying conditions that a task, state, authority, or evidence must satisfy.
- evaluation receiptAn evaluation receipt is a record of a model's training and generator versions, held-out workload, evaluation results, and release decision.
- agent loopAn iterative agent-control pattern in which an agent perceives, reasons, acts, and reflects, with subsequent steps selected dynamically.
- guardrail sandwichAn execution pattern that places a state-changing action between trusted pre-check and post-check controls, with a controlled executor in between.
- planningAn agent capability for decomposing goals and selecting an intended sequence of steps toward achieving them.
- execution topologyA structural arrangement that determines how steps or agents are organized and executed in an agent system.
- iteration limitA loop limit is the maximum number of iterations permitted in a loop.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
