liveBook · Manning

What you will learn in Designing AI Systems

A guide to production-ready platforms

by Suhas Suresha and Dewang Sultania · Manning Early Access Program (MEAP)

Chapter by chapter

Chapter 1: Why your AI projects need a platform

This chapter introduces why moving AI prototypes into production requires shared platform infrastructure rather than one-off application code.

1.1 The AI Wild West: why winging it doesn't scale · 1.1.1 The production wake-up call · 1.1.2 The universal pattern: prototype vs. production · 1.1.3 What if the infrastructure already existed? · 1.1.4 AI sprawl · 1.2 The AI infrastructure reality: the model is just 2% of the story · 1.2.1 Traditional ML Customer Support (2015) · 1.2.2 Modern GenAI Customer Support (2026) · 1.3 Thinking from first principles: what AI applications actually need · 1.4 Your AI platform blueprint: what we're building · 1.5 Platform in action · 1.6 Why platform thinking changes everything · 1.7 Summary

What you learn in chapter 1 →

Chapter 2: Building with the platform: SDK and API design

This chapter introduces the SDK and API contracts that turn the platform’s services into productive developer and consumer interfaces.

2.1 Designing the ideal developer experience · 2.1.1 Immediate productivity through effortless setup · 2.1.2 The conversation memory challenge · 2.1.3 The organizational knowledge problem · 2.1.4 The safety imperative · 2.1.5 The external system integration challenge · 2.1.6 The optimization dilemma · 2.1.7 The Multi-Step Coordination Challenge · 2.1.8 Synthesis: the developer experience contract · 2.2 Code to container: the deployment story · 2.2.1 The shared process trap · 2.2.2 Choosing the right isolation mechanism · 2.2.3 One workflow, one service · 2.2.4 Managing execution · 2.3 Exposing workflows as APIs · 2.3.1 The API sprawl problem · 2.3.2 Unified workflow exposure · 2.3.3 The communication protocol · 2.3.4 Patterns of AI interaction · 2.4 The communication architecture · 2.4.1 The entry point problem · 2.4.2 The internal communication challenge · 2.4.3 Tracing the Complete Flow · 2.4.4 Why these decisions scale · 2.5 Building the SDK · 2.5.1 Connection to the API Gateway · 2.5.2 Providing access to services · 2.5.3 Capturing deployment information · 2.5.4 The SDK foundation · 2.6 Summary

What you learn in chapter 2 →

Chapter 3: The Model Service: Your platform’s gateway to AI models

This introduction positions the Model Service as the platform component that turns application requests into AI responses while hiding provider and operational complexity.

3.1 The model service contract · 3.1.1 Generating responses · 3.1.2 Discovering available models · 3.1.3 Managing system prompts · 3.1.4 Registering custom models · 3.1.5 The gRPC contract · 3.1.6 Request and response structures · 3.2 Provider abstraction · 3.2.1 How providers differ · 3.2.2 The unified provider interface · 3.2.3 OpenAI message format as platform standard · 3.2.4 What adapters do · 3.2.5 The adapter implementation pattern · 3.3 Streaming responses · 3.3.1 The streaming architecture · 3.3.2 The ChatChunk message · 3.3.3 Streaming and error handling · 3.4 Resilience: fallbacks and retries · 3.4.1 Retry configuration · 3.4.2 Fallback configuration · 3.4.3 Configuration examples · 3.5 Routing strategies · 3.5.1 Routing configuration · 3.5.2 Cost-aware routing · 3.5.3 Load-based routing · 3.5.4 Feature-based routing · 3.5.5 Combining patterns · 3.6 Rate limiting · 3.7 Caching for cost and performance · 3.7.1 Two levels of caching · 3.7.2 The cache interface · 3.7.3 Monitoring cache effectiveness · 3.8 Observability: Cost tracking and metrics · 3.8.1 What the model service tracks · 3.8.2 Feeding the observability service · 3.8.3 Enabling informed decisions · 3.9 Integrating with the SDK · 3.9.1 The ModelClient · 3.9.2 Method implementation pattern · 3.9.3 Streaming support · 3.9.4 The complete picture · 3.10 Summary

What you learn in chapter 3 →

Chapter 4: The Session Service: Teaching your AI to remember

This chapter introduces the Session Service as the platform component that preserves conversational context, stores it as structured session data, and prepares it within model context-window limits.

4.1 What a session contains · 4.2 The session service contract · 4.3 Storage abstraction · 4.3.1 Choosing the right database · 4.3.2 The storage interface · 4.4 Implementing the Session Service backend · 4.4.1 The Database Schema · 4.4.2 The Storage Implementation · 4.4.3 The gRPC service implementation · 4.5 Integrating with the SDK · 4.6 Model-managed memory · 4.6.1 Where memory lives · 4.6.2 The memory model · 4.6.3 Extending the storage interface for memories · 4.6.4 SDK Integration · 4.6.5 A complete workflow example · 4.7 Managing context windows · 4.7.1 The token budget problem · 4.7.2 Simple truncation · 4.7.3 Compressing history with summarization · 4.7.4 Hierarchical memory · 4.7.5 Retrieval-augmented memory · 4.7.6 Putting it together · 4.8 Summary

What you learn in chapter 4 →

Chapter 5: The Data Service: teaching AI what your organization knows

This chapter introduces the Data Service as platform infrastructure that grounds AI responses in searchable organizational knowledge rather than relying only on conversational memory.

5.1 From documents to searchable knowledge · 5.2 Indexes: organizing knowledge · 5.2.1 Why isolation matters · 5.2.2 Index configuration · 5.2.3 Index operations · 5.3 Ingestion pipeline: from raw files to vectors · 5.3.1 The challenge of diverse formats · 5.3.2 Pipeline architecture · 5.3.3 Format detection and text extraction · 5.3.4 Metadata: the filtering foundation · 5.3.5 Chunking: breaking text into retrievable pieces · 5.3.6 Generating embeddings · 5.3.7 Document lifecycle · 5.3.8 Document management · 5.3.9 Asynchronous ingestion · 5.4 Vector storage and search · 5.4.1 Vector store interface · 5.4.2 Choosing a vector store backend · 5.4.3 The pgvector implementation · 5.4.4 Search orchestration · 5.5 Hybrid search: combining vectors with keywords · 5.5.1 Adding keyword search to the platform · 5.5.2 PostgreSQL keyword search implementation · 5.5.3 Merging results: Reciprocal Rank Fusion · 5.5.4 Putting it together · 5.6 Service contract and complete retrieval flow · 5.7 Summary

What you learn in chapter 5 →

Chapter 6: Tools and Guardrails: Enabling safe, managed AI actions

This introduction explains why AI actions need platform-managed tools and guardrails, and outlines the services that provide them.

6.1 Reframing tools as platform-managed capabilities · 6.1.1 The tool service contract · 6.1.2 Tool definitions beyond function schemas · 6.1.3 The SDK interface · 6.2 The tool registry: namespacing and discovery · 6.2.1 Capability-based discovery · 6.2.2 Version control · 6.3 Tool execution: adapters, credentials and reliability · 6.3.1 The adapter pattern · 6.3.2 Credential isolation · 6.3.3 The credential store interface · 6.3.4 Synchronous and asynchronous execution · 6.4 Interoperability: the Model Context Protocol (MCP) · 6.4.1 From NxM to N+M, and then to M · 6.4.2 The architecture of MCP: hosts, clients and servers · 6.4.3 The three MCP primitives · 6.4.4 What MCP does not cover · 6.4.5 Integrating MCP with the platform · 6.5 Tool execution safeguards · 6.5.1 Resource limits · 6.5.2 Circuit breakers · 6.6 Guardrails: not just filters, but execution policies · 6.7 Guardrails in practice: input, output and behavioral policies · 6.7.1 Input guardrails: protecting the system · 6.7.2 Output guardrails: the last line of defense · 6.7.3 Behavioral guardrails: controlling actions and logic · 6.8 Declarative policy and observability · 6.8.1 Policy as configuration · 6.8.2 Making safety visible · 6.9 Summary

What you learn in chapter 6 →

Chapter 7: The Observability Service: seeing what your AI does

This introduction frames the Observability Service as the platform component that unifies telemetry across services so teams can understand AI behavior, cost, performance, and quality.

7.1 Why AI systems need specialized observability · 7.1.1 From infrastructure health to output quality · 7.1.2 Cross-service correlation · 7.1.3 Cost, and the bridge to experimentation · 7.2 The observability data model · 7.2.1 Deriving the model from questions · 7.2.2 How the primitives connect: tracing a single request · 7.2.3 Logs and metrics: complementing the request-level model · 7.3 The Observability Service contract · 7.3.1 Walking through the contract · 7.3.2 Ingestion vs query patterns · 7.4 Structured logging, metrics, and distributed tracing · 7.4.1 Structured logging for AI-specific debugging · 7.4.2 Metrics collection · 7.4.3 Distributed tracing and the debugging workflow · 7.5 How platform services report telemetry · 7.5.1 TracedService: automatic instrumentation · 7.5.2 Domain-specific telemetry · 7.5.3 The observability client: batching and buffering · 7.5.4 Adding custom observability beyond the defaults · 7.6 Quality scores and cost attribution · 7.6.1 Scores: measuring response quality · 7.6.2 Cost attribution and budget tracking · 7.7 Putting it all together · 7.8 Summary

What you learn in chapter 7 →

Chapter 8: The Experimentation Service: improving what your AI does

This chapter introduction positions the Experimentation Service as the infrastructure that turns observability signals into a continuous, controlled loop for improving AI behavior.

8.1 The Experimentation Service contract · 8.2 Target lifecycle and evaluation · 8.3 Evaluation: datasets, offline scoring, online scoring, and human review · 8.3.1 Datasets: the foundation of evaluation · 8.3.2 Offline evaluation: comparing variants before deployment · 8.3.3 Online evaluation: scoring production traffic in real-time · 8.3.4 Annotation queues: calibrating with human judgment · 8.4 A/B testing infrastructure · 8.5 Putting it all together · 8.6 Summary

What you learn in chapter 8 →

Chapter 9: The Workflow Service: orchestrating and deploying AI applications

This introduction positions the Workflow Service as the platform component that turns developer workflow code into scalable HTTP applications by separating deployment operations from request-time execution.

9.1 Workflow runtime server · 9.1.1 The @workflow decorator · 9.1.2 From decorated function to HTTP server · 9.1.3 Synchronous mode · 9.1.4 Streaming mode · 9.1.5 Asynchronous mode · 9.2 Async job lifecycle · 9.2.1 Job storage and the polling endpoint · 9.2.2 Progress reporting · 9.3 Production readiness: concurrency, reliability, and health · 9.3.1 Concurrency inside a single container · 9.3.2 Retry at the network layer · 9.3.3 Health endpoints · 9.4 Workflow composition · 9.4.1 Calling other workflows · 9.4.2 Parallel workflow calls · 9.4.3 Response mode handling · 9.5 The Workflow Service contract · 9.5.1 Registry operations · 9.5.2 Deployment operations · 9.6 Deployment pipeline · 9.6.1 What genai-platform deploy does · 9.6.2 Route registration with the API gateway · 9.7 Runtime management · 9.8 Summary

What you learn in chapter 9 →

Chapter 10: Building an AI assistant: putting the platform to work

This introduction frames the chapter as an incremental build of Claw, a complete AI assistant that composes every platform service while keeping application code focused on assistant behavior.

10.1 The blueprint · 10.2 The simplest assistant: a model call in a workflow · 10.3 Teaching Claw to remember conversations · 10.4 Long-term memory: remembering across sessions · 10.5 Grounding Claw in reality: knowledge retrieval and agentic RAG · 10.6 From conversation to action: tools and the agent loop · 10.7 Safety: guardrails at every step · 10.8 The prompt that makes it all work: context engineering · 10.9 The complete agent loop · 10.9.1 Streaming the response · 10.10 Seeing what Claw does: observability · 10.11 Making Claw better: experimentation · 10.12 Deploying Claw · 10.13 What the platform gave us · 10.14 Summary

What you learn in chapter 10 →

Concepts explained in this book

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.