Chapter 10: Building an AI assistant: putting the platform to work
This introduction frames the chapter as an incremental build of Claw, a complete AI assistant that composes every platform service while keeping application code focused on assistant behavior.
After this chapter you will be able to
- Platform service roles. Describe the role each platform service plays in Claw: models generate responses, sessions remember conversations, data retrieves organizational knowledge, tools call external systems, guardrails enforce safety, observability tracks behavior, experimentation measures improvement, and workflows deploy and scale the application.
- Incremental assistant construction. Explain the chapter's construction approach: begin with the simplest assistant, expose its limitations, and add a platform service connection and capability at each step until every service participates in handling one user message.
- Claw request architecture. Trace the path of a user message through the complete Claw architecture, from a Slack, Discord, or web interface through the API Gateway and workflow container to a streamed response.
- Platform-orchestrated assistant capabilities. Describe the capabilities the Claw workflow coordinates: session and long-term memory, knowledge retrieval, context assembly, model generation, tool execution, guardrail validation, exchange storage, observability, and experiment variants. Explain that application code calls platform methods instead of rebuilding those capabilities.
- Minimal assistant architecture. Describe the simplest Claw assistant as a workflow function that accepts a user message and wraps one Model Service call to return the model’s generated response.
- Platform-managed integration behavior. Explain how the SDK and Model Service keep the application code short by handling request formatting, provider resolution, retries, and provider fallback outside the Claw workflow.
- Stateless conversation limitation. Explain why the initial Claw cannot hold a conversation: each message arrives independently, and the Model Service receives only the current message and system prompt, with no memory of earlier sessions or exchanges.
- Session-backed conversation continuity. Explain how Claw uses the Session Service to make a model behave as though it remembers earlier turns within one conversation. Describe loading a session's history into the model context on each request and storing the new exchange afterward.
- Sliding-window context tradeoff. Explain why extended session histories can exceed a model's context window and how a sliding window limits that risk. State that it retains the most recent N messages in model context while preserving older history in storage, at the cost of Claw forgetting early details during the current response.
- Cross-session memory architecture. Explain how long-term memory differs from current-session conversation history and how the Session Service provides both to Claw. Describe the role of persistent per-user memories in allowing a later session to continue with known preferences, projects, and working context.
- Selective memory tool use. Describe how Claw gives the model a memory tool so the model decides, from conversational context and the tool description, which information is worth persisting. Distinguish durable preferences or project details from incidental information that should not be stored.
- Memory context injection. Trace how each request retrieves the user's memory as a dictionary and injects formatted facts in a second system message alongside the main system prompt and conversation history. Explain that the prompt directs Claw to use these facts naturally rather than announce that it remembered them.
- Basic RAG retrieval flow. Describe how basic retrieval-augmented generation grounds Claw: the workflow searches the Data Service on every request, adds top retrieved documents to the model context alongside conversation history and user memory, and then generates an answer.
- Agentic RAG decisions. Distinguish agentic RAG from pipeline-based RAG by explaining that the model receives knowledge search as a tool and uses conversational context to decide whether to search, formulate or reformulate a query, choose an index, and repeat searches when results are insufficient.
- Tool registration and managed execution. Describe how Claw registers action tools through the Tool Service and how the workflow executes requested calls through platform.tools.execute. Explain that registrations make tools discoverable to the model and that execution differs for handler-backed tools versus endpoint and MCP tools.
- Credential and behavior metadata. Explain why tool registrations use credential_ref rather than raw secrets, and describe how is_read_only and is_idempotent metadata affect safety decisions and retries. State why failed non-idempotent actions should not be automatically retried without explicit confirmation.
- Bounded agent reasoning cycle. Trace the agent loop in which the model receives the accumulated conversation, either returns final text or requests one or more tools, the workflow appends tool results to message history, and the model reasons again. Explain that the iteration limit bounds costs and unwanted repeated actions when the model is confused.
- Defense-in-depth guardrail layers. Explain how Claw applies defense-in-depth through input validation at the gateway, tool-argument validation before each tool execution, and output filtering before a response reaches the user.
- Guardrail-aware agent-loop handling. Describe how the agent loop invokes input and output guardrails at its boundaries and intercepts every proposed tool call for argument validation; explain why a blocked tool action is returned to the model as a tool result rather than raised as an exception.
- Context engineering discipline. Explain context engineering as the application-level discipline of deciding what information enters the model context, what is trimmed, and what is sacrificed when the context window is limited.
- Context sources and separation. Describe the six context sources for Claw and distinguish the static system prompt from dynamic, request-time context. State that user memory and conversation history come from the Session Service, retrieved documents from the Data Service, tool descriptions from the Tool Service, the system prompt from the Prompt Registry, and the current message from the user.
- Budget allocation and ordering. Explain Claw's application-specific allocation and ordering strategy: it gives 40 percent of the remaining dynamic budget to documents because grounding is its main differentiator, uses the rest for conversation history trimming oldest messages first, and places the system prompt first, grounding early, history in the middle, and the current message last.
- Complete agent loop flow. Trace the complete Claw request flow from an incoming message through guardrails, session and memory loading, context assembly, model decisions, possible tool-use iterations, output filtering, storage, and the returned response. Explain that tool results loop back to the model until it produces final text or reaches the iteration limit.
- Agentic branching decisions. Distinguish an agent loop from a fixed pipeline by describing the branching decisions around model output and tool execution. State that the middle loop—model tool calls, observed results, and continued reasoning—is the core agentic behavior.
- Streaming agent-loop control. Explain how Claw streams each agent-loop turn by calling chat_stream instead of chat, yielding text tokens as they arrive while buffering tool-call deltas until the stream ends. Describe how the finish reason determines whether the loop ends with text or executes tools and continues.
- Streamed workflow event model. Describe the two event types delivered by the streaming workflow: token events for model text and status events announcing that a tool is about to run. Explain why tool-call JSON is kept internal and why status events prevent apparent dead air during tool execution.
- Incremental output guardrails. Explain why synchronous filter_output cannot safely be used after streamed tokens have already reached the client, and describe how filter_output_stream incrementally filters tokens before yielding them. State that it buffers short spans, typically about a sentence, to detect violations before release, trading a small time-to-first-token delay for no unfiltered output.
- Automatic cross-service tracing. Describe how Claw platform calls are automatically instrumented by Chapter 7 gRPC middleware when they cross service boundaries, with telemetry flowing into the Observability Service. State the service-specific data captured for model, data, tool, and guardrail calls and how a trace tree represents their parent-child relationships.
- Trace-driven performance diagnosis. Explain how Sarah uses per-span latency and cost breakdowns to identify the highest-leverage optimization target and distinguish internal work from an external dependency. Use the example that the two model calls dominate the request latency, while vector search and guardrails contribute much less, and that the calendar call can be managed with timeout and fallback behavior.
- Complementary quality scoring. Distinguish the automated retrieval-relevance scorer from the model-based judge scorer, including why they run independently. Explain that the automated rule reads the Data Service relevance score at no extra model cost and runs on every trace, whereas the judge samples 10% of traffic to control model-evaluation cost while retaining statistically useful data.
- Observability and experimentation roles. Distinguish observability from experimentation in Claw's improvement process: observability reveals what is happening, while experimentation determines whether a proposed change actually makes the assistant better.
- Prompt A/B test design. Describe Sarah's A/B test of a tools-first prompt against the current prompt: users are split evenly for seven days, remain stickily assigned to a variant, and the platform compares the variants across tool selection, quality, cost, and latency.
- Statistical promotion decision. Explain how trace-level variant tags, confidence intervals, and p-values support a promotion decision: Sarah promotes the new prompt version only if it improves tool selection without harming quality, cost, or latency; otherwise she reverts while the control remains in production.
- Workflow deployment contract. Describe how the @workflow decorator acts as the deployment contract between Claw's code and the Workflow Service, specifying the workflow name, API path, scaling parameters, resource requirements, timeout, and retry policy. Trace what happens when Sarah runs platform deploy.
- Stateless horizontal scaling. Explain how the Workflow Service packages Claw and the SDK runtime into a container image, creates a Kubernetes deployment, registers /claw/chat with the API gateway, and configures autoscaling from two to ten replicas. Explain why stateless workflow containers and externally stored platform state make horizontal scaling possible.
- Asynchronous workflow composition. Describe the fan out / fan in pattern for complex research: Claw starts child workflows that retrieve from multiple knowledge bases and the web in parallel, then performs a synthesis call to the Model Service. Explain why this work uses async mode, returning a job ID for client polling, and why independent child containers prevent research bursts from starving the conversational workflow.
- Platform-enabled production assistant. Explain how Claw grew from a seven-line workflow wrapping one model call into a production-grade assistant with roughly two hundred lines of application code because platform services addressed each production concern.
- Production capabilities provided. Describe the production capabilities Claw obtains from the platform: memory within and across sessions, grounded answers from organisational documents, managed actions with credentials and rate limits, three-layer safety, request-to-token tracing, and statistically controlled improvement.
- Compounding platform reuse. Explain why the platform investment compounds: later assistants reuse the same services and add only application-specific logic, even when their service combinations differ.
What it builds on
- The model service contract (chapter 3) · The Model Service contract for generating responses.
- The session service contract (chapter 4) · The Session Service contract for conversation memory.
- Service contract and complete retrieval flow (chapter 5) · The Data Service retrieval flow.
- Reframing tools as platform-managed capabilities (chapter 6) · The Tool Service contract for managed capabilities.
- What a session contains (chapter 4) · What session state and memory contain.
- The tool registry: namespacing and discovery (chapter 6) · Managed tool namespacing and capability discovery.
- Tool execution: adapters, credentials and reliability (chapter 6) · Credential isolation and reliable tool execution.
- Tool execution safeguards (chapter 6) · Tool safeguards such as resource limits and circuit breakers.
Section by section
Introduction
This introduction frames the chapter as an incremental build of Claw, a complete AI assistant that composes every platform service while keeping application code focused on assistant behavior.
- Platform service roles: Describe the role each platform service plays in Claw: models generate responses, sessions remember conversations, data retrieves organizational knowledge, tools call external systems, guardrails enforce safety, observability tracks behavior, experimentation measures improvement, and workflows deploy and scale the application.
Why it matters: Claw works as a complete assistant because it composes specialized platform services rather than rebuilding their infrastructure in application code. - Incremental assistant construction: Explain the chapter's construction approach: begin with the simplest assistant, expose its limitations, and add a platform service connection and capability at each step until every service participates in handling one user message.
Why it matters: Building capability by capability makes the reason for each architectural component visible rather than treating an agent as a single opaque system. - Platform-managed infrastructure: Explain why Claw can remain relatively small application code: the application specifies what the assistant should do, while platform services provide persistent and scalable memory, managed tool discovery, credential isolation, rate limiting, configurable safety policies, distributed tracing, and quality scoring.
Why it matters: This separation lets a developer deliver an assistant without embedding the operational machinery in every application.
Key terms: Model Service, Session Service, Data Service, Tool Service, Guardrails Service, Observability Service, Claw, agent loop, platform service, persistent memory
10.1 The blueprint
This section previews Claw’s complete architecture, showing a small workflow orchestrating platform services and the incremental path to assembling it.
- Claw request architecture: Trace the path of a user message through the complete Claw architecture, from a Slack, Discord, or web interface through the API Gateway and workflow container to a streamed response.
Why it matters: This gives you the end-to-end map for understanding how each later enhancement fits into the assistant. - Platform-orchestrated assistant capabilities: Describe the capabilities the Claw workflow coordinates: session and long-term memory, knowledge retrieval, context assembly, model generation, tool execution, guardrail validation, exchange storage, observability, and experiment variants. Explain that application code calls platform methods instead of rebuilding those capabilities.
Why it matters: The blueprint illustrates the platform’s central value: application developers compose managed capabilities rather than implement their infrastructure from scratch. - Incremental construction roadmap: Explain the section’s incremental build approach: each following section identifies a limitation in the current Claw version and addresses it with a service built earlier, until the complete architecture is wired up by section 10.9.
Why it matters: This roadmap helps you understand why the assistant is built in stages rather than introduced as one large implementation.
Key terms: API Gateway, Claw workflow container, workflow function, context window, Model Service, streaming, session, long-term memory, knowledge base, tool calls
10.2 The simplest assistant: a model call in a workflow
This section builds Claw’s minimal workflow-wrapped model call, shows what the platform handles for it, and identifies its lack of conversational memory.
- Minimal assistant architecture: Describe the simplest Claw assistant as a workflow function that accepts a user message and wraps one Model Service call to return the model’s generated response.
Why it matters: This establishes the small application-level starting point from which the assistant gains memory, knowledge, tools, and safety features. - Platform-managed integration behavior: Explain how the SDK and Model Service keep the application code short by handling request formatting, provider resolution, retries, and provider fallback outside the Claw workflow.
Why it matters: The platform lets application authors express assistant intent rather than rebuilding model-provider integration and reliability logic. - Prompt and workflow deployment integration: Describe the two integration choices in the minimal assistant: the system prompt is registered with the Model Service so it can be versioned and changed without redeployment, and the @workflow decorator turns the function into a containerized HTTP endpoint with operational metadata.
Why it matters: These choices separate changeable assistant instructions and operational deployment concerns from the assistant’s business logic. - Stateless conversation limitation: Explain why the initial Claw cannot hold a conversation: each message arrives independently, and the Model Service receives only the current message and system prompt, with no memory of earlier sessions or exchanges.
Why it matters: An assistant users trust for scheduling and task management needs continuity, so this stateless behavior is a credibility problem that motivates adding memory.
Key terms: workflow function, Model Service, model call, Claw, SDK, provider resolution, retries, fallbacks, request formatting, system prompt
10.3 Teaching Claw to remember conversations
This section gives Claw within-session conversational continuity by persisting exchanges through the Session Service while managing finite model context windows.
- Session-backed conversation continuity: Explain how Claw uses the Session Service to make a model behave as though it remembers earlier turns within one conversation. Describe loading a session's history into the model context on each request and storing the new exchange afterward.
Why it matters: This turns isolated model calls into a natural multi-turn conversation that can handle follow-up questions. - Session identification and creation: Describe the role of the session_id parameter and the get_or_create pattern in reconnecting a client to a conversation. State that get_or_create creates and returns a session for a first-time user, but returns the existing session on later messages using the same session identifier.
Why it matters: A stable session identifier lets a client resume the correct conversation instead of starting over on every request. - Sliding-window context tradeoff: Explain why extended session histories can exceed a model's context window and how a sliding window limits that risk. State that it retains the most recent N messages in model context while preserving older history in storage, at the cost of Claw forgetting early details during the current response.
Why it matters: Keeping all stored history is not the same as being able to provide all of it to a model with a limited context window.
Key terms: Session Service, persistent session, conversation history, context window, exchange, session_id, get_or_create, PostgreSQL, sliding window, most recent N messages
10.4 Long-term memory: remembering across sessions
This section gives Claw persistent, model-managed user memory across sessions by selectively storing facts and injecting them into each request's context.
- Cross-session memory architecture: Explain how long-term memory differs from current-session conversation history and how the Session Service provides both to Claw. Describe the role of persistent per-user memories in allowing a later session to continue with known preferences, projects, and working context.
Why it matters: Persistent memory makes an assistant feel like an ongoing working relationship rather than a stateless question-answering tool. - Selective memory tool use: Describe how Claw gives the model a memory tool so the model decides, from conversational context and the tool description, which information is worth persisting. Distinguish durable preferences or project details from incidental information that should not be stored.
Why it matters: Selective, model-directed storage captures useful working context without turning every utterance into permanent memory. - Key-value memory updates: Explain why persistent memories are stored as named key-value facts and how this lets newer information overwrite an existing value rather than create contradictory entries. Give an example such as updating preferred_meeting_time when a user's preference changes.
Why it matters: Named keys keep accumulated user memory current and manageable as details change over time. - Memory context injection: Trace how each request retrieves the user's memory as a dictionary and injects formatted facts in a second system message alongside the main system prompt and conversation history. Explain that the prompt directs Claw to use these facts naturally rather than announce that it remembered them.
Why it matters: Stored facts improve responses only when the workflow places them in the model's context in a useful, unobtrusive form.
Key terms: Session Service, conversation history, long-term memories, model-managed memory, current session, memory tool, Tool Service, save_fact, tool description, prompt engineering
10.5 Grounding Claw in reality: knowledge retrieval and agentic RAG
This section grounds Claw in organizational knowledge by progressing from fixed retrieval-augmented generation to model-directed, agentic RAG.
- Basic RAG retrieval flow: Describe how basic retrieval-augmented generation grounds Claw: the workflow searches the Data Service on every request, adds top retrieved documents to the model context alongside conversation history and user memory, and then generates an answer.
Why it matters: This gives the assistant factual organizational context instead of relying on plausible but unverified model guesses. - Retrieval relevance and attribution: Explain why basic RAG applies a similarity-score threshold and attaches source attribution to retrieved chunks. State that a threshold avoids injecting irrelevant documents into the token budget, while document IDs let Claw cite knowledge-base claims for user verification.
Why it matters: Relevant, attributable retrieval reduces confused answers and lets users verify where factual claims came from. - Agentic RAG decisions: Distinguish agentic RAG from pipeline-based RAG by explaining that the model receives knowledge search as a tool and uses conversational context to decide whether to search, formulate or reformulate a query, choose an index, and repeat searches when results are insufficient.
Why it matters: Agentic retrieval handles context-dependent follow-up questions that a raw-message search query cannot answer well. - Knowledge-base ingestion preparation: State that Claw can search only after the knowledge base is ingested up front, and explain why semantic chunking preserves meaningful policy clauses and code examples better than naïve fixed-token chunking.
Why it matters: Retrieval quality depends on preparing documents as coherent searchable chunks before Claw receives user requests.
Key terms: retrieval-augmented generation (RAG), Data Service, platform.data.search, context window, knowledge base, score threshold, semantic match, token budget, source attribution, document_id
10.6 From conversation to action: tools and the agent loop
This section equips Claw to act through managed tools using a bounded agent loop that reasons, executes, observes results, and continues until it can answer.
- Tool registration and managed execution: Describe how Claw registers action tools through the Tool Service and how the workflow executes requested calls through platform.tools.execute. Explain that registrations make tools discoverable to the model and that execution differs for handler-backed tools versus endpoint and MCP tools.
Why it matters: Managed tool registration lets an assistant take useful actions without embedding every integration directly in application code. - Credential and behavior metadata: Explain why tool registrations use credential_ref rather than raw secrets, and describe how is_read_only and is_idempotent metadata affect safety decisions and retries. State why failed non-idempotent actions should not be automatically retried without explicit confirmation.
Why it matters: Action tools can create real external consequences, so credentials and retry behavior must be controlled by the platform rather than improvised in application code. - Bounded agent reasoning cycle: Trace the agent loop in which the model receives the accumulated conversation, either returns final text or requests one or more tools, the workflow appends tool results to message history, and the model reasons again. Explain that the iteration limit bounds costs and unwanted repeated actions when the model is confused.
Why it matters: The bounded reason-act-observe loop is what enables multi-step agent behavior while preventing unlimited tool use. - Dynamic tool discovery and interoperability: Describe how build_model_tools converts all tools registered under the claw namespace into model-usable definitions, so newly registered tools are available on the next request without changing the agent-loop code. Explain how an MCP-compatible web-search provider can be replaced without changing Claw's code.
Why it matters: Dynamic discovery and MCP interoperability let the assistant gain or replace capabilities without repeatedly rewriting its orchestration logic. - Structured tool-error recovery: Explain that tool failures are returned as structured tool results instead of being thrown as unhandled errors, allowing the model to explain a failure and continue reasoning about alternatives. Illustrate this with a calendar conflict that Claw can report and work around with different times.
Why it matters: Returning failures to the model as usable context lets the assistant recover gracefully rather than ending the conversation at the first integration problem.
Key terms: Tool Service, platform.tools.execute, handler-backed tools, endpoint tools, MCP tools, tool registration, credential_ref, Credential Store, is_read_only, is_idempotent
10.7 Safety: guardrails at every step
This section makes Claw safe through layered input, tool-argument, and output guardrails integrated into its agent loop.
- Defense-in-depth guardrail layers: Explain how Claw applies defense-in-depth through input validation at the gateway, tool-argument validation before each tool execution, and output filtering before a response reaches the user.
Why it matters: Multiple safety layers ensure that one missed unsafe condition does not become an unsafe outcome. - Policy-specific failure prevention: Describe the concrete safety policies configured for Claw and the failure modes they address: topic blocking for disallowed requests, calendar and ticket constraints for tool calls, and PII redaction and hedging for outputs.
Why it matters: Specific policies turn general safety goals into enforceable protections against predictable assistant failures. - Guardrail-aware agent-loop handling: Describe how the agent loop invokes input and output guardrails at its boundaries and intercepts every proposed tool call for argument validation; explain why a blocked tool action is returned to the model as a tool result rather than raised as an exception.
Why it matters: Returning a constraint as part of the conversation lets the assistant explain the limit and adapt rather than failing abruptly. - Transparent output modifications: Explain that output filters record their changes in applied_filters and return them as safety_flags so the frontend can tell users when a response was modified for safety.
Why it matters: Visible safety modifications build more trust than silently changing what the assistant said.
Key terms: Guardrails Service, defense-in-depth, input validation, tool argument validation, output filtering, topic policy, block action, calendar.create_event, tickets.create, session-scoped limit
10.8 The prompt that makes it all work: context engineering
This section explains how Claw assembles competing static and dynamic information sources into a token-budgeted, application-specific model context.
- Context engineering discipline: Explain context engineering as the application-level discipline of deciding what information enters the model context, what is trimmed, and what is sacrificed when the context window is limited.
Why it matters: The information available to the model determines whether it stays grounded, follows the conversation, and uses tokens effectively. - Context sources and separation: Describe the six context sources for Claw and distinguish the static system prompt from dynamic, request-time context. State that user memory and conversation history come from the Session Service, retrieved documents from the Data Service, tool descriptions from the Tool Service, the system prompt from the Prompt Registry, and the current message from the user.
Why it matters: Separating stable instructions from changing request data keeps the reviewed system prompt versioned and avoids unnecessary prompt updates. - Budget allocation and ordering: Explain Claw's application-specific allocation and ordering strategy: it gives 40 percent of the remaining dynamic budget to documents because grounding is its main differentiator, uses the rest for conversation history trimming oldest messages first, and places the system prompt first, grounding early, history in the middle, and the current message last.
Why it matters: Deliberate allocation and placement preserve Claw's most valuable grounding while accounting for models' weaker attention to material in the middle of the context.
Key terms: context engineering, context window, token budget, user memory, conversation history, retrieved documents, tool descriptions, system prompt, Prompt Registry, docs-versus-history trade-off
10.9 The complete agent loop
This section assembles Claw’s previously built capabilities into a complete, platform-backed agent loop and distinguishes the engineering layers around the model.
- Complete agent loop flow: Trace the complete Claw request flow from an incoming message through guardrails, session and memory loading, context assembly, model decisions, possible tool-use iterations, output filtering, storage, and the returned response. Explain that tool results loop back to the model until it produces final text or reaches the iteration limit.
Why it matters: This flow shows how separate platform services become an agent that can reason, act, and respond safely rather than a one-shot model call. - Agentic branching decisions: Distinguish an agent loop from a fixed pipeline by describing the branching decisions around model output and tool execution. State that the middle loop—model tool calls, observed results, and continued reasoning—is the core agentic behavior.
Why it matters: The branching loop is what lets Claw adapt its next action to the model’s decisions and tool results. - Harness engineering layers: Describe the difference among prompt engineering, context engineering, and harness engineering in Claw. Explain that the harness is everything around the model—such as the agent loop, tool routing, guardrails, memory, retrieval, observability, retries, and fallbacks—and that platform services reduce this composition layer to roughly a hundred lines rather than thousands.
Why it matters: This vocabulary clarifies that a useful AI assistant depends on orchestration and platform capabilities, not only on the wording of its prompt.
Key terms: agent loop, input guardrails, context assembly, tool calls, iteration limit, output filtering, agent, pipeline, branching decisions, agentic behavior
10.9.1 Streaming the response
This subsection adapts the Claw agent loop to stream safe, user-facing progress and response tokens while preserving its existing tool execution and platform responsibilities.
- Streaming agent-loop control: Explain how Claw streams each agent-loop turn by calling chat_stream instead of chat, yielding text tokens as they arrive while buffering tool-call deltas until the stream ends. Describe how the finish reason determines whether the loop ends with text or executes tools and continues.
Why it matters: This lets an assistant feel responsive without changing the underlying agent-loop behavior. - Streamed workflow event model: Describe the two event types delivered by the streaming workflow: token events for model text and status events announcing that a tool is about to run. Explain why tool-call JSON is kept internal and why status events prevent apparent dead air during tool execution.
Why it matters: Users need visible feedback during tool work without being exposed to implementation details. - Incremental output guardrails: Explain why synchronous filter_output cannot safely be used after streamed tokens have already reached the client, and describe how filter_output_stream incrementally filters tokens before yielding them. State that it buffers short spans, typically about a sentence, to detect violations before release, trading a small time-to-first-token delay for no unfiltered output.
Why it matters: Streaming must retain the same output-safety guarantee as a completed synchronous response. - Platform ownership of streaming safety: Justify keeping incremental output filtering in the Guardrails Service rather than reimplementing it in each workflow, while leaving Claw's workflow focused on composing context, tools, and service calls. Describe which existing workflow responsibilities remain unchanged when streaming is added.
Why it matters: Central platform capabilities keep difficult safety behavior consistent across assistants while workflows remain simpler.
Key terms: chat_stream, chat, agent loop, tool_call_delta, finish reason, token events, status events, ephemeral indicators, filter_output, filter_output_stream
10.10 Seeing what Claw does: observability
This section shows how Claw’s automatic platform telemetry and quality scoring make production behavior measurable and actionable.
- Automatic cross-service tracing: Describe how Claw platform calls are automatically instrumented by Chapter 7 gRPC middleware when they cross service boundaries, with telemetry flowing into the Observability Service. State the service-specific data captured for model, data, tool, and guardrail calls and how a trace tree represents their parent-child relationships.
Why it matters: Automatic tracing lets operators investigate a complete assistant request without adding custom instrumentation. - Trace-driven performance diagnosis: Explain how Sarah uses per-span latency and cost breakdowns to identify the highest-leverage optimization target and distinguish internal work from an external dependency. Use the example that the two model calls dominate the request latency, while vector search and guardrails contribute much less, and that the calendar call can be managed with timeout and fallback behavior.
Why it matters: Breaking a request into measured spans prevents teams from optimizing minor components while the dominant source of delay or cost remains unchanged. - Complementary quality scoring: Distinguish the automated retrieval-relevance scorer from the model-based judge scorer, including why they run independently. Explain that the automated rule reads the Data Service relevance score at no extra model cost and runs on every trace, whereas the judge samples 10% of traffic to control model-evaluation cost while retaining statistically useful data.
Why it matters: Operational metrics show what happened, but complementary quality scores reveal whether Claw’s responses are actually good and where quality is failing. - Quality pattern discovery: Explain how quality scores accumulated over time expose recurring problems that isolated operational metrics cannot show, such as weak question categories, user-specific tool failures, quality degradation in longer conversations, and differences among knowledge-base indexes. State that these patterns form the evidence base for the next improvement loop.
Why it matters: Long-term score patterns turn scattered request data into prioritized, evidence-based improvements.
Key terms: gRPC middleware, Observability Service, trace tree, span, Model Service, Data Service, latency, cost breakdown, caching, external dependency
10.11 Making Claw better: experimentation
This section shows how Claw uses a statistically controlled A/B test to decide whether a prompt change improves production behavior.
- Observability and experimentation roles: Distinguish observability from experimentation in Claw's improvement process: observability reveals what is happening, while experimentation determines whether a proposed change actually makes the assistant better.
Why it matters: Seeing a problem in production is not enough to know that a fix will improve the system. - Prompt A/B test design: Describe Sarah's A/B test of a tools-first prompt against the current prompt: users are split evenly for seven days, remain stickily assigned to a variant, and the platform compares the variants across tool selection, quality, cost, and latency.
Why it matters: A controlled production comparison prevents an apparently obvious prompt improvement from causing unnoticed regressions. - Statistical promotion decision: Explain how trace-level variant tags, confidence intervals, and p-values support a promotion decision: Sarah promotes the new prompt version only if it improves tool selection without harming quality, cost, or latency; otherwise she reverts while the control remains in production.
Why it matters: Statistical evidence and a live control let the team improve Claw safely instead of relying on intuition about prompt changes.
Key terms: Observability Service, Experimentation Service, quality scores, A/B test, tools-first prompt, current prompt, sticky assignment, sample-size guarantees, control group, variant
10.12 Deploying Claw
This section deploys Claw through the Workflow Service and shows how stateless scaling, asynchronous jobs, and parallel child workflows support production research tasks.
- Workflow deployment contract: Describe how the @workflow decorator acts as the deployment contract between Claw's code and the Workflow Service, specifying the workflow name, API path, scaling parameters, resource requirements, timeout, and retry policy. Trace what happens when Sarah runs platform deploy.
Why it matters: The deployment contract turns application code into a managed, reachable production workflow. - Stateless horizontal scaling: Explain how the Workflow Service packages Claw and the SDK runtime into a container image, creates a Kubernetes deployment, registers /claw/chat with the API gateway, and configures autoscaling from two to ten replicas. Explain why stateless workflow containers and externally stored platform state make horizontal scaling possible.
Why it matters: Keeping state in platform services lets Claw scale containers without losing conversations, memory, knowledge, or credentials. - Asynchronous workflow composition: Describe the fan out / fan in pattern for complex research: Claw starts child workflows that retrieve from multiple knowledge bases and the web in parallel, then performs a synthesis call to the Model Service. Explain why this work uses async mode, returning a job ID for client polling, and why independent child containers prevent research bursts from starving the conversational workflow.
Why it matters: Async composition keeps long-running research responsive while isolating its scaling needs from interactive chat.
Key terms: @workflow decorator, deployment contract, platform deploy, Workflow Service, container image, Kubernetes deployment, /claw/chat, autoscaler, replicas, stateless
10.13 What the platform gave us
This section concludes that a shared AI platform turns a small model-call workflow into a production-grade assistant while making future AI applications cheaper to build through service reuse.
- Platform-enabled production assistant: Explain how Claw grew from a seven-line workflow wrapping one model call into a production-grade assistant with roughly two hundred lines of application code because platform services addressed each production concern.
Why it matters: The platform supplies reusable production capabilities so application code can focus on assistant-specific logic. - Production capabilities provided: Describe the production capabilities Claw obtains from the platform: memory within and across sessions, grounded answers from organisational documents, managed actions with credentials and rate limits, three-layer safety, request-to-token tracing, and statistically controlled improvement.
Why it matters: These capabilities distinguish a dependable production assistant from an uncontrolled model prototype. - Compounding platform reuse: Explain why the platform investment compounds: later assistants reuse the same services and add only application-specific logic, even when their service combinations differ.
Why it matters: Reuse lets a team build its next assistant without recreating the infrastructure that made the first one production-ready.
Key terms: Claw, workflow, model call, production-grade assistant, platform service, application-specific logic, sessions, organisational documents, managed credentials, rate limits
10.14 Summary
This summary shows how Claw combines the platform’s services into a production-ready AI assistant, with context engineering, safe agent actions, and built-in improvement capabilities.
Concepts explained in this chapter
- ReActAn agent loop is an iterative execution pattern in which a model selects tools, observes their results, and continues reasoning until it produces a final response or reaches a stopping condition.
- context engineeringContext engineering is the deliberate assembly of the information supplied to a model call within constraints such as token capacity and attention behavior.
- content filteringThe detection or suppression of inappropriate, harmful, or otherwise disallowed content in model inputs or outputs.
- sliding windowA sliding window is a context-management technique that retains the most recent items while storing older items outside the active processing window.
- AI agentAn agent is an AI system composed of a model and execution harness that can select and perform actions toward a goal.
- agentic RAGAgentic RAG is a retrieval-augmented generation approach in which a model decides when and how to retrieve information and can iterate on searches.
- nlp-based chunkingSemantic chunking detects topic boundaries by comparing embeddings of consecutive sentences and starting chunks where semantic similarity drops sharply.
- retrieval-augmented generationA retrieval-augmented generation pipeline retrieves relevant context and supplies it to a generative model when producing a response.
- retrieve-then-generate pipelinePipeline-based RAG is a retrieval-augmented generation approach in which retrieval runs as a fixed mechanical step for each request.
- conversational aiA personal AI assistant is an AI application that helps an individual by understanding requests, providing information, and performing actions on the user's behalf.
- memoryModel-managed memory is a capability in which an AI model selects, stores, and retrieves information from persistent memory outside its immediate context window.
- organizational knowledgeOrganizational knowledge is the body of policies, procedures, guides, directories, and other information maintained by an organization.
- validationInput validation is the process of checking incoming requests or data against defined validity, safety, and policy requirements before processing.
- PII scrubbingPII redaction is the removal or masking of personally identifiable information from generated or transmitted content.
- defense in depthDefense in depth is a security pattern that applies multiple independent protective controls so that failure of one control does not eliminate protection.
- tool registrationTool registration is the declaration of an AI-accessible tool together with its invocation details, credentials, and behavioral metadata.
- tool resultA tool result is the structured output returned by an invoked tool and appended to the conversation for subsequent model reasoning.
- tool usetool_calls is a message field containing an assistant's requested tool or function invocations.
- qualityA score is a numerical or categorical value representing how well an observed operation or output performed against an evaluation criterion.
- spanA span is a distributed-tracing record representing one timed operation within a trace and its relationship to parent and child operations.
- workflow decoratorA workflow decorator is a function annotation that converts an ordinary function into a managed workflow process.
- stream processingA streaming response is an output delivery mode that sends generated content incrementally as it becomes available instead of waiting for the complete result.
- trajectoryA trace is a correlated record of a request's journey through a system, including its operations, timing, metadata, and nested work units across components.
- automated scoringAn automated scorer is a non-generative evaluation procedure that computes response-quality scores from recorded metrics or rules.
- observability serviceA service that collects and exposes operational information about application behavior, including latency, usage, and performance.
- relevanceA retrieval relevance distribution is the statistical distribution of relevance scores produced for retrieved results across queries.
- lost in the middleThe lost in the middle phenomenon is a language-model behavior in which information positioned in the middle of a long context receives less attention than information near the beginning or end.
- dialogueA conversation is a structured sequence of messages exchanged between a user, an assistant, and the system instructions governing the interaction.
- context windowA context window is the bounded amount of input and prior interaction content that a language model can process for one generation.
- sessionA session is a structured record representing one conversation between a user and an AI application, including its messages and identifying metadata.
- session serviceA session service is a platform service that stores and manages conversational state, user preferences, interactions, and conversation history across sessions.
- autoscalingAuto-scaling is the automatic adjustment of computing resources in response to workload demand.
- child workflowA child workflow is a workflow invoked by another workflow and executed as an independently managed deployment.
- response modeA response mode is a workflow configuration that determines whether results are returned synchronously or incrementally as a stream.
- workflow compositionWorkflow composition is the pattern of building an application workflow by invoking other independently deployed workflows instead of duplicating their logic.
- workflow serviceA workflow service is a platform service that coordinates sequential and parallel operations, state management, retries, and failure recovery in complex processes.
- model serviceA model service is a platform service that provides a unified interface to multiple AI model providers while handling provider-specific APIs, routing, resilience, and operational concerns.
- promptAn instruction or input supplied to an AI model to elicit a response or behavior.
- A/B testA controlled experiment that compares alternative variants by assigning them to different user or traffic groups and measuring outcomes.
- platformA software and infrastructure foundation that provides shared capabilities for building and operating applications.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
