Chapter 8: Deploying agents and agentic systems

Introduction

The introduction frames deployment as a choice among ways to make an agent available, from embedding it in an application to using containers.

• Agent deployment options: Describe how deploying or productizing an agent can range from embedding it in an existing application to using containers for a more scalable and controlled approach.
  Why it matters: This distinction helps you think about deployment as a choice suited to the agent’s needs.

Key terms: deploy, productize, embedding, containers

8.1 Strategies for consuming agents

This section introduces ways to consume agents, explains when browser embedding is appropriate, and connects backend deployment to containerized microservices.

• Agent consumption patterns: Distinguish an agent embedded in an application from one accessed through a microservice API or consumed by another agent through MCP or A2A. Explain that how an agent is consumed can influence how it is deployed.
  Why it matters: The way an application or another agent uses your agent guides the deployment choice.
• Browser-side deployment constraints: Explain the key exposure, CORS, and rate-limit problems that arise when agent logic runs in a browser.
  Why it matters: A browser prototype can appear simple while exposing credentials or failing under real users.
• Appropriate agent placement: Justify when browser-side agent logic is reasonable and when an authenticated backend service is a better fit.
  Why it matters: Choosing placement by use case avoids treating a workable prototype as a safe production design.
• Containerized microservice rationale: Describe how separating code into concerns leads to container-hosted microservices and why agents can fit that approach.
  Why it matters: This explains why a separate agent service can be easier to change and deploy than agent code tied to an application.

Key terms: embedded, microservice, API, MCP, A2A, browser-side agent deployment, API keys, CORS, rate limits, embed pattern

8.1.1 Embedding real-time voice agents into web applications

This section shows how a real-time voice agent can run in a browser, walks through a minimal demo, and explains how to evaluate its AI-generated deployment code.

• Browser voice agent design: Explain why a real-time voice agent can be embedded in client-side JavaScript and why its tools and backend work should be kept responsive.
  Why it matters: A voice interface feels responsive only if the agent's work does not introduce substantial delays.
• Voice demo operation and limits: Describe what the single-file browser demo provides, how a reader connects its microphone, and what the agent cannot do in this version.
  Why it matters: The demo illustrates browser voice interaction without presenting its connection and capabilities as production-ready.
• Generated scaffold engineering review: Explain why this chapter uses AI-generated deployment scaffolds and what review remains necessary before adopting generated code.
  Why it matters: Generated scaffolds can accelerate deployment work, but they still need the same engineering judgment as hand-written code.

Key terms: real-time voice agents, client-side JavaScript, latency, OpenAI Agents SDK Realtime package, ephemeral key, Connect mic, MCP servers, vibe coding, deployment scaffolds, engineering judgment

8.1.2 Hosting agents through an API

This section shows how to expose an existing image-generation agent through a FastAPI endpoint, explains its request and response choices, and describes how to run and check the service.

• Agent as an API service: Explain how the example makes an existing agent available to HTTP clients without changing the underlying agent code.
  Why it matters: The same wrapper pattern can make an existing agent or tool accessible to other applications.
• Request and image response choices: Justify the use of POST and describe the image format returned by the generate endpoint, including the stated base64 alternative.
  Why it matters: These choices affect how a client sends a description and handles the generated image.
• Service run and output check: Describe how to start and call the example service and what result confirms that image generation completed.
  Why it matters: Running the service and inspecting its output confirms that the API-backed agent produces an image.

Key terms: FastAPI, POST endpoint, agent web service, POST, GET, generate endpoint, array of bytes, base64, uvicorn, VS Code

8.1.3 Consuming an agent web service in a web application

Shows how a browser-based voice agent delegates image generation to a backend agent while remaining interactive, and distinguishes the example from a production deployment.

• Interactive image generation flow: Trace how the browser-based RealTimeAgent uses generateImageTool to hand an image request to a backend image agent while continuing to talk with the user.
  Why it matters: This pattern lets an agent keep a conversation going while a slow backend task runs.
• Example startup requirements: Describe what must be running or configured before using the browser application to request an image by voice.
  Why it matters: The browser example needs both its access key and the backend image agent before the voice request can produce an image.
• Production infrastructure beyond example: Explain what infrastructure a real image-generation deployment needs around the agent-side architecture and why the example does not implement those production concerns.
  Why it matters: The working demonstration illustrates an agent pattern, not everything required to serve real traffic reliably.

Key terms: RealTimeAgent, generateImageTool, proxy, image agent, ephemeral key, backend API agent, queue, worker pool, temporary storage, notification mechanism

8.2 Dockerizing agent systems

This section explains why containerized microservices suit agent systems, how a frontend agent can use them, and where the containers can be hosted.

• Focused agent microservices: Explain why the section recommends containerizing agent API applications as microservices, especially in a multi-agent system.
  Why it matters: This explains the deployment benefit of keeping agents focused and replaceable.
• Frontend access to containerized agents: Describe how the browser-based frontend agent uses containerized agents and what can be placed in a container.
  Why it matters: It shows how containerized agents can remain available to an application without requiring every container to have the same contents.
• Container hosting options: Distinguish the hosting options named for a containerized agent and identify which one the book covers here.
  Why it matters: It separates the available deployment choices from the local deployment the book actually demonstrates.

Key terms: microservices architecture, agent API applications, subgoal, frontend agent, API interface, MCP server, multi-agent workflows, Docker Desktop, Kubernetes, web/API application

8.2.1 Containerizing an agent microservice

This section walks through generating, building, running, testing, and locally managing a containerized agent microservice.

• Agent microservice container workflow: Describe how the section moves from an agent web application to a locally running container: generate and review a Dockerfile, build an image with Docker Desktop running, run it with an OpenAI API key, and test the service.
  Why it matters: This is the path from application code to a locally testable agent service, not a production-ready image by default.
• Local container management: Explain how keeping a named container makes it manageable in Docker Desktop and what developers can do there while experimenting locally.
  Why it matters: Keeping a container available in Docker Desktop makes local inspection and troubleshooting easier.

Key terms: Dockerfile, Docker Desktop, image, container, OpenAI API key, curl, --name image-gen, logs, CI/CD

8.2.2 Orchestrating agentic systems with Docker Compose

This section shows how Docker Compose defines and runs a local multi-container agent system, then notes a limit of using Compose for cloud deployment.

• Compose orchestration advantage: Explain why Docker Compose is more practical than deploying the agent microservices individually.
  Why it matters: This is the reason the example uses Compose rather than separate container deployments.
• Compose file service definitions: Describe the format and basic organization of the example Compose file and what its services represent.
  Why it matters: The service definitions connect the Compose file to the containers it orchestrates.
• Local stack run sequence: Trace how a reader can run and access the example agent stack locally.
  Why it matters: These steps turn the Compose definition into a locally usable agent application.
• Cloud portability limit: Distinguish running the Compose deployment locally from deploying its agent containers to a cloud provider.
  Why it matters: A working local Compose stack is not automatically a cloud deployment.

Key terms: Docker Compose, multi-agent stack, microservices, YAML, Compose version, service definitions, realtime-voice-web, realtime-web-agent, realtime-image-agent, Docker Desktop

8.2.3 Externalizing local agent microservices

Explains how a tunnel can briefly expose local agent services for testing or demos, how to access them with localtunnel, and why production use calls for cloud deployment.

• External tunnels for local agents: Explain that services such as ngrok and localtunnel create a public endpoint that routes traffic to a locally running container, making them useful for webhook tests, integration debugging, and brief demos.
  Why it matters: A tunnel is handy when an outside user or service needs to reach something still running on your machine.
• Tunnel exposure and deployment boundary: Explain the security risks of leaving a tunnel open and justify choosing cloud deployment for customer access, sustained traffic, or anything beyond a brief demo.
  Why it matters: A convenient demo link should not become an unprotected production entry point to your local machine.
• Localtunnel access walkthrough: Describe the section's localtunnel walkthrough: with Node.js installed, expose a local port, open the resulting URL, pass the password page, and use the web agent voice page.
  Why it matters: The walkthrough shows what temporary access to the running agent looks like in practice.

Key terms: tunneling services, external endpoints, ngrok, localtunnel, authentication, rate limiting, cloud deployment, security boundaries, access control, Node.js

8.3 Considering advanced deployment strategies

Introduces the deployment decisions that become important when moving from a proof of concept to a dependable multi-agent system.

8.3.1 Choosing a runtime: Edge, API, or event-driven

This section explains how latency and workload shape the choice between edge, synchronous API, and event-driven worker runtimes for agents.

• Latency as a deployment guide: Explain why the desired user experience helps determine where an agent runs: interactive agents need low latency, while agents with little user interaction can tolerate longer waits.
  Why it matters: Start with how quickly the user needs a response before choosing where to run the agent.
• Three agent runtime options: Distinguish edge, synchronous API, and event-driven worker deployments by where or how they run and the workloads each suits.
  Why it matters: These distinctions let you match the runtime to the interaction rather than use one deployment style for every agent.
• Simplest suitable runtime: Apply the section’s rule of choosing the simplest runtime that meets latency and tool requirements, including when to shift fast interactions to edge or heavy work to workers.
  Why it matters: You can begin with a straightforward deployment and change only the parts whose requirements call for it.

Key terms: latency, user experience, real-time deployment, Edge (browser/mobile), Synchronous API microservice, Event-driven worker agents, queue, simplest runtime, latency target, tool requirements

8.3.2 The three “wires” of communication

Compares three communication channels for agents and tools and explains how latency and deployment needs guide the choice.

• Three communication wires: Distinguish WebRTC/WebSocket, HTTP + SSE, and message buses by the interactions each supports and where each fits.
  Why it matters: The channel determines whether an agent interaction feels immediate, streams a response, or lets slow work happen in the background.
• Latency and deployment choice: Explain why the communication wire is selected according to latency and deployment requirements, including the local MCP connection alternative.
  Why it matters: Choosing by requirements avoids treating one transport as the default for every agent or tool connection.

Key terms: WebRTC/WebSocket (Realtime), HTTP + SSE (MCP), message bus, barge-in, wire, latency, deployment requirements, STDIO, HTTP+SSE

8.3.3 Practical multi-agent topologies that adapt well

The section presents a front-door multi-agent topology that accommodates different communication and latency needs while keeping most work and control inside worker agents.

• Adaptive front-door topology: Explain how the front-door agent pattern extends an orchestrator-like pattern to accommodate differing communication and latency requirements across the front-door agent and backend workers.
  Why it matters: A user-facing agent need not impose one communication approach on workers with different latency needs.
• Worker-local control and workflows: Describe where the section recommends placing work and control, and name the kinds of workflows a worker agent can contain.
  Why it matters: Putting coordination where the work happens prevents the front-door agent from becoming unnecessarily complex.

Key terms: front-door agent pattern, orchestrator pattern, backend worker agent, latency, worker agent, flow, orchestrator, collaboration, front-door agent

8.3.4 State, memory, and idempotency

This section distinguishes short-term conversational state from long-term knowledge and explains why idempotent, cacheable tool calls matter when agents retry or replay work.

• Short-term and long-term memory: Distinguish where a deployed agentic system typically stores conversation turns from how it accesses long-term facts and knowledge.
  Why it matters: Separating conversation turns from longer-term knowledge helps keep state manageable in large, chatty systems.
• Idempotent tool calls and caching: Explain what makes a tool call idempotent and why agents benefit from using idempotent keys to cache its results.
  Why it matters: Agents may retry work, so cacheable, idempotent calls make replay faster without repeating side effects.

Key terms: conversational memory, long-term facts, semantic search indexes, vector stores, idempotent, idempotent key, caching, replay, duplicate side effects

8.3.5 Release engineering for agents (prompts, tools, models)

Treat agents as software by versioning their components, releasing changes through gates, and recording the exact model and tool endpoints used for each turn.

• Versioned agent components: Explain which agent components should be versioned and why prompts might be versioned separately from code.
  Why it matters: An agent release includes more than its application code.
• Gated release promotion: Describe the staged path for promoting an agent release and what should happen if service objectives degrade.
  Why it matters: Staged promotion builds confidence before an agent change reaches everyone.
• Per-turn model and tool records: Explain what model and tool information to record for each agent turn and why.
  Why it matters: When an agent response causes a problem, the exact dependencies used for that turn help reconstruct it.

Key terms: version everything, prompts, tool schemas, tool servers, safety switches, model selections, promote with gates, offline tests, shadow traffic, miniature canary

8.3.6 Observability matters

This section explains how traces, metrics, and logs make a production agent observable so its behavior and outcomes can be diagnosed.

• End-to-end agent traces: Explain how tracing turns and tool calls across the UI, gateway, agent, tools, and model helps locate activity and diagnose problems. Describe how correlation IDs and Phoenix metadata make traces findable.
  Why it matters: Finding the activity behind a particular turn or tool call gives you a place to start diagnosing failures.
• Production metric categories: Distinguish operational, quality, and product metrics using examples from the section, and explain why operational health alone does not show whether the agent is useful.
  Why it matters: A fast, error-free system may still produce poor answers or fail the task it was deployed to accomplish.
• Structured diagnostic logs: Describe what logs add alongside traces and the precautions the section recommends when capturing them.
  Why it matters: Logs can expose useful debugging context that traces do not capture, without requiring you to retain personal information.

Key terms: trace spans, correlation IDs, session_id, turn_id, tool_call_id, Phoenix, operational metrics, quality metrics, product metrics, structured JSON logs

8.3.7 Reliability patterns: Timeouts, fallbacks, and budgets

This section explains how time budgets, fallbacks, and circuit breakers help agents handle failures without disrupting users.

• Critical path time budgets: Explain that each critical path should have a time budget enforced by the caller as an SLA, with a fallback when a call exceeds it.
  Why it matters: A slow tool need not leave the user waiting indefinitely.
• Failure fallbacks and circuit breakers: Distinguish a fallback for a failed or over-budget tool call from a circuit breaker that responds to repeated tool failures.
  Why it matters: These patterns keep a failing tool from turning every affected request into a failure.
• User-facing graceful degradation: Explain how an agent can keep serving users in a reduced form rather than exposing errors, and why that matters for confidence in the system.
  Why it matters: A useful reduced response is more trustworthy than repeatedly exposing users to failures.

Key terms: critical path, time budget, SLA, fallback, fallbacks, circuit breakers, shed load, graceful degradation, operational confidence

8.3.8 Cost control and model routing

This section explains how to control production agent costs through value-based measurement, model routing, context trimming, and deliberate caching without sacrificing output quality.

• Cost Relative to Value: Explain why agent cost should be judged against the value of the task it replaces or produces, and describe what to measure before optimizing.
  Why it matters: The cheapest agent is not necessarily the one that delivers the best value.
• Intent Routing Tradeoffs: Describe how intent routing sends requests to downstream models or agents, and justify evaluating routing accuracy before replacing a frontier router with a cheaper small model.
  Why it matters: Savings from a cheap router can disappear if it sends requests down the wrong path.
• Context Trimming Tradeoffs: Describe practical ways to reduce context tokens and explain why each trimming decision needs quality evaluation.
  Why it matters: Less context saves tokens on repeated calls, but missing context can make an agent less effective.
• Reusable Prompt Prefixes: Explain how prompt caching saves input costs and how to arrange stable and changing prompt content to obtain cache hits.
  Why it matters: Prompt layout determines whether repeated instructions actually receive the caching discount.
• Selective Result Caching: Distinguish good and poor candidates for caching, and describe how to manage the savings-versus-staleness tradeoff.
  Why it matters: A cache that saves calls but serves outdated or contextually wrong results is not a successful optimization.
• Token Cost and Latency: Explain why token-reducing cost optimizations can also change agent response latency and deployment choices.
  Why it matters: A cost optimization may also make an agent faster for its users.

Key terms: cost-to-value ratio, cost per session, cost per resolved task, cost per user, routing by intent, frontier model, routing accuracy, downstream model or agent, context trimming, compressed history

8.4 Security, safety, and governance in production

The section explains why agents’ ability to act autonomously makes security and safety essential production concerns.

• Autonomy and safety priorities: Explain why security and safety must be first-class concerns for agents: agents can act autonomously.
  Why it matters: An agent’s ability to act makes security and safety important from the outset.

Key terms: agents, autonomously, security, safety, first-class concerns

8.4.1 A quick threat model for agentic systems

This section shows how to map production-agent assets to exposed surfaces and risks, then explains prompt injection and the layered defenses it requires.

• Asset Based Threat Modeling: Explain how to build a threat model by identifying assets, mapping them to exposed surfaces and likely risks, and prioritizing the highest risks rather than relying on a security checklist.
  Why it matters: Mapping assets to exposures helps you focus protection where it addresses a real risk.
• Production Attack Surface Exposures: Describe how different production surfaces put different assets or capabilities at risk, using examples from the client, gateway, agent runtime, tool servers, model provider, or storage.
  Why it matters: A defense is more useful when it matches the assets and capabilities exposed at that part of the system.
• Prompt Injection Sources And Mechanism: Explain how instructions embedded in content an agent reads can be mistaken for authoritative instructions, and distinguish direct user-origin injection from indirect injection in third-party content.
  Why it matters: An agent can follow an attacker’s directions even when the legitimate user did nothing wrong.
• Layered Prompt Injection Defenses: Explain why prompt-injection defenses must be layered, especially why tool outputs should be treated as untrusted data, alongside validation, output filtering, and human checkpoints for high-stakes actions.
  Why it matters: Layering controls reduces the chance that one missed malicious instruction becomes a data leak or harmful action.

Key terms: assets, attack surfaces, threat model, client, gateway/API, agent runtime, tool servers, model provider, storage, prompt injection

8.4.2 Identity and access for people, services, and agents

This section explains how to limit agent access to a user’s permissions, protect browser credentials, and record who uses agents.

• User scoped agent access: Explain why an agent acting for a user should have only that user’s limited access, and how user authorization must carry through the agent’s tool and service calls, including RAG document access.
  Why it matters: An agent’s tools should not let it reach data or services that its user cannot access.
• Browser agent credential handling: Describe how real-time browser agents obtain short-lived client secrets without exposing provider API keys in the browser.
  Why it matters: This lets a browser agent connect without exposing a provider API key to the client.
• Agent user activity records: State that user access to and usage of agents should be logged, and identify user objects in tracing as a way to do it.
  Why it matters: User-linked traces make agent access and usage visible.

Key terms: user authentication, user authorization, tools, MCP servers, RAG systems, ephemeral client secrets, provider API keys, TTL, user objects, tracing

8.4.3 Secrets and configuration management

The section gives production practices for handling secrets and limiting access to tools and resources.

• Runtime secret handling: Explain where secrets should be kept, how they should be provided at runtime, and how they should be handled afterward.
  Why it matters: These practices limit where production secrets are stored and exposed.
• Least privilege for resources: Describe how to scope users' access to tools and other resources.
  Why it matters: Scoped permissions keep access limited to what a user needs.

Key terms: container images, runtime, environment variables, secret manager, rotate secrets, scope, least privilege, tools, resources

8.4.4 Tool safety: Sandboxing and egress control

This section explains why agent tools need restricted access and how to secure tools exposed through MCP servers.

• Risks of agent tool use: Explain why tools must be treated as potentially unsafe even when no attack is apparent.
  Why it matters: This is why tool restrictions are necessary even for agents you intend to behave well.
• Restricted tool execution and access: Describe the restrictions recommended for a tool's execution, filesystem access, network access, and resource consumption.
  Why it matters: These limits reduce what a failed or misused tool can reach or consume.
• Security of MCP tool servers: Explain what to check and enforce when accessing tools through an MCP server.
  Why it matters: Securing the tool itself is not enough if its server or connection is inadequately protected.

Key terms: abusive tool use, hallucinatory agent/LLM, Sandbox, Filesystem, ephemeral, outbound allowlist, internet egress, Resource limits, MCP servers, HTTPS

8.4.5 Prompt-injection and data-exfiltration defenses

This section explains how to reduce prompt-injection risk for user-facing agents through instruction, tool, and execution safeguards while recognizing the limits of prompt rules.

• Untrusted input and instruction hierarchy: Explain why direct user input must be treated as untrusted and how system prompts should handle instructions embedded in that input.
  Why it matters: Treating user content as instructions can make an agent vulnerable to injection and jailbreak attacks.
• Constrained tools and execution: Describe how schemas, typed output, restricted execution, and tool allowlists reduce the ways untrusted content can affect an agent.
  Why it matters: These constraints reduce the opportunities for malicious input to become a tool call or executable action.
• Proportionate prompt safety rules: Explain how to choose prompt safety rules for an agent's actual interaction surfaces and why those rules cannot be the only defense.
  Why it matters: Matching rules to real risks avoids unnecessary prompt overhead without mistaking prompt instructions for guaranteed protection.

Key terms: untrusted, instruction hierarchy, system prompts, injection, jailbreak attacks, schema-first tools, JSON Schemas, input/output sanitation, allowlists, user authorization

8.4.6 Safety and policy enforcement

This section explains how to enforce production-agent safety and organizational policies through controls that can be checked independently of the agent.

• External policy enforcement: Explain why production policies should be enforced in front of or alongside an agent rather than only through its prompt.
  Why it matters: A prompt can guide an agent, but it is not a dependable or auditable policy boundary.
• Content and data safeguards: Describe the content-safety and data-privacy obligations that production agents need to cover.
  Why it matters: Safe-looking output alone does not address unsafe inputs or improper handling of private data.
• Accountability and usage controls: Describe how audit records, rate limits, and authorization address different production-agent risks.
  Why it matters: These controls make agent actions traceable while limiting overuse and unauthorized access.
• Effective human approval checkpoints: Explain the four design decisions that make human-in-the-loop approval useful for high-risk agent actions.
  Why it matters: A bare approval button may invite rubber-stamping and does not protect a system when review stalls or is bypassed.
• Early policy planning: Justify applying safety and policy guidelines before production deployment.
  Why it matters: Planning boundaries early helps contain unpredictable agent behavior before it becomes a production failure.

Key terms: policy enforcement, external enforcement, prompt injection, content safety, data privacy and compliance, content filters, data residency, audit and traceability, rate limiting and abuse prevention, access control and authorization

8.5 Exercises

Five hands-on exercises walk the reader from a browser-only voice agent to an API-backed, containerized multi-agent stack.

8.6 Summary

The summary connects deployment choices, agent architecture, and production safeguards to moving agents from demos to dependable systems.