liveBook · Manning

Chapter 2: Building with the platform: SDK and API design

What you will learn in chapter 2 of Designing AI Systems

This chapter introduces the SDK and API contracts that turn the platform’s services into productive developer and consumer interfaces.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This chapter introduces the SDK and API contracts that turn the platform’s services into productive developer and consumer interfaces.

Key terms: SDK contract, platform services, AI workflows, infrastructure complexity, API contract, streaming responses, authentication, synchronous, asynchronous, developer productivity

2.1 Designing the ideal developer experience

This section frames the platform's SDK design around making AI application development effortless while leaving important behavior under developer control.

Key terms: developer experience, SDK interface, infrastructure complexity, patient intake assistant, conversation memory, company policies, medical advice, healthcare logic, infrastructure code, complex parts invisible

2.1.1 Immediate productivity through effortless setup

This subsection establishes that the platform should let developers create a working AI assistant in a few lines of code by hiding infrastructure complexity behind sensible defaults.

Key terms: immediate value, minimal setup, effortless simplicity, sensible defaults, Model Service, provider authentication, request formatting, response parsing, error handling, cost tracking

2.1.2 The conversation memory challenge

This section defines a developer-friendly Session Service interface that gives AI workflows automatic conversational memory without requiring developers to manage storage or context themselves.

Key terms: conversation memory, Session Service, session storage, conversation serialization, context window management, get_or_create(), get_messages(), add_messages(), sensible defaults, context window limits

2.1.3 The organizational knowledge problem

This section defines the need for a Data Service that lets AI applications use current organizational knowledge through simple, reliable semantic search.

Key terms: organizational knowledge, model's training data, Data Service, knowledge integration, platform.data.search(), document processing, vector embeddings, semantic search, relevance ranking, accurate, current information

2.1.4 The safety imperative

This section establishes configurable, platform-enforced guardrails as the way to keep healthcare AI interactions safe without requiring every workflow to implement safety logic.

Key terms: Guardrails Service, technical controls, platform capability, model safety, healthcare-specific policies, no_medical_advice, healthcare_compliance, add_policy, configuration, input validation

2.1.5 The external system integration challenge

This section defines a platform approach for integrating external systems: register tools declaratively and let the platform securely coordinate their use in model conversations.

Key terms: external tools, Tool Service, tool registration, parameters, operations, credential, API key, credential isolation, tool execution, model calls

2.1.6 The optimization dilemma

This section explains why prompt choices should be optimized through systematic evaluation rather than personal intuition.

Key terms: prompt optimization, evaluation setup, test data, expected answers, metrics, benchmark

2.1.7 The Multi-Step Coordination Challenge

This section shows how a platform workflow coordinates AI services into a reliable, observable, and composable multi-step application pipeline.

Key terms: Session Service, Data Service, Model Service, Guardrails, registered tools, workflow, @workflow decorator, managed process, error recovery, automatic retries

2.1.8 Synthesis: the developer experience contract

This section synthesizes the developer experience contract: a unified platform should hide distributed AI infrastructure so developers can build sophisticated applications as naturally as business logic.

Key terms: developer experience contract, immediate productivity, conversation memory, knowledge integration, safety controls, tool integration, Model Service, Session Service, Data Service, Guardrails Service

2.2 Code to container: the deployment story

This section frames the deployment questions and basic runtime requirements that arise after a developer defines a Python workflow with the @workflow decorator.

Key terms: @workflow decorator, workflow, Model Service, frontend application, deployment

2.2.1 The shared process trap

This section shows why running all AI workflows in one shared process fails and establishes infrastructure-enforced, process-level isolation as the minimum requirement.

Key terms: shared process, workflow, memory leak, infinite loop, resource needs, process-level isolation, own environment, logical separation, infrastructure, workflow isolation

2.2.2 Choosing the right isolation mechanism

This section compares processes, virtual machines, and containers for workflow isolation and selects containers as the practical balance of security, resource efficiency, and operational needs.

Key terms: plain processes, virtual machines, VMs, containers, isolation, resource overhead, container, packaged environment, dependencies, process isolation

2.2.3 One workflow, one service

This section establishes that every workflow should be an independently deployable service to avoid coordination burdens and enable clean operational ownership.

Key terms: one workflow equals one independently deployable service, @workflow decorator, deployment lifecycle, one-to-one mapping, independent deployment, independent scaling, independent versioning, traffic shifting, resource attribution, cost tracking

2.2.4 Managing execution

This section explains how the Workflow Service centrally operates deployed workflow containers so developers can focus on business logic rather than production infrastructure.

Key terms: Workflow Service, registry, deployed workflows, container images, resource requirements, replica counts, lifecycle management, health monitoring, health checks, replicas

2.3 Exposing workflows as APIs

This section frames the need for a clear external API contract for deployed workflows, separating client communication transport from AI-specific interaction timing patterns.

Key terms: workflow, deployable service, external consumers, clear contract, transport mechanism, interaction patterns, stream results, AI operations, interface, frontend team

2.3.1 The API sprawl problem

This section defines API sprawl as inconsistent workflow interfaces and establishes standardized workflow exposure as the platform response.

Key terms: API sprawl, workflows, Workflow Service, custom integration code, authentication scheme, external clients, standardized way to expose workflows, accessible endpoints, communication protocol

2.3.2 Unified workflow exposure

This section defines how workflows receive stable developer-chosen API paths under unified domains so external access stays consistent and avoids API sprawl.

Key terms: workflow name, API path, api_path, public contract, external clients, unified domain structure, domain, logical boundary, security boundary, authentication mechanisms

2.3.3 The communication protocol

This section selects HTTP as the universal protocol for workflow access and explains why AI workflows require multiple HTTP interaction patterns for different execution times.

Key terms: HTTP, HTTP methods, POST, GET, headers, status codes, request-response APIs, interaction patterns, streaming responses, token-by-token

2.3.4 Patterns of AI interaction

This section defines synchronous, asynchronous, and streaming HTTP interaction patterns for exposing AI workflows according to their duration and result-delivery needs.

Key terms: synchronous requests, asynchronous requests, streaming responses, interaction patterns, POST, GET, JSON, object storage, synchronous interaction, job identifier

2.4 The communication architecture

This section frames the distinct external and internal communication problems that the platform must solve before tracing a request through the system.

Key terms: external communication, internal communication, workflows, platform services, trust boundaries, public internet, datacenter, network, Sarah's assistant, workflow container

2.4.1 The entry point problem

This section explains why an API Gateway is the platform's stable entry point and how it centralizes reliable request routing for independently deployed workflow containers.

Key terms: API Gateway, single-entry point, workflow, containers, replicas, stable URL, authentication, rate limiting, encrypted connections, load balancing

2.4.2 The internal communication challenge

This section explains why platform services use gRPC internally while external clients communicate through HTTP, balancing universal access with efficient service-to-service calls.

Key terms: HTTP, gRPC, external clients, internal services, API gateway, SDK, Protocol Buffers, schema, generated client code, Session Service

2.4.3 Tracing the Complete Flow

This section traces one user request through the gateway, workflow, and platform services to show how centralized HTTPS and gRPC routing enables efficient, scalable, observable, and secure communication.

Key terms: API gateway, workflow, Session Service, Data Service, Model Service, HTTPS, gRPC, Protocol Buffer, TLS termination, gRPC proxy

2.4.4 Why these decisions scale

This section explains how the gateway-based communication architecture scales traffic, services, deployments, and internal gRPC calls without forcing workflow developers to manage distributed-systems complexity.

Key terms: API gateway, horizontal scaling, load balancer, stateless, service registry, independent scaling, load profiles, replicas, healthy instances, health checking

2.5 Building the SDK

This section introduces the SDK as the bridge that turns developer-written Python workflows into coordinated calls to distributed platform services and deployable workflow metadata.

Key terms: SDK, workflow code, distributed platform services, gRPC, GenAIPlatform object, runtime entrypoint, gateway connection logic, service-specific clients, decorator, deployment information

2.5.1 Connection to the API Gateway

This section explains how the SDK platform object connects isolated workflow containers to platform services through an API gateway, using environment-aware gateway configuration.

Key terms: API gateway, gRPC, Session Service, service containers, client object, GenAIPlatform, gateway URL, GENAI_GATEWAY_URL, localhost, constructor

2.5.2 Providing access to services

This section shows how the SDK exposes distributed platform services as local-looking clients that are lazily created and routed through a central gRPC API Gateway.

Key terms: GenAIPlatform, service client, SessionClient, ModelClient, DataClient, gRPC, lazy initialization, @property, _sessions, API Gateway

2.5.3 Capturing deployment information

This section shows how the SDK’s @workflow decorator attaches deployment metadata to a Python function and how the deployment tool uses that metadata to build, publish, and register a workflow.

Key terms: @workflow decorator, wrapper function, _workflow_metadata, deployment configuration, workflow name, API path, replica counts, CPU threshold, auto-scaling, response mode

2.5.4 The SDK foundation

This section establishes the SDK pattern that gives developers uniform access to platform services and hides distributed infrastructure behind simple client calls and deployment commands.

Key terms: *Client, platform gateway, x-target-service, gRPC calls, routing metadata, GenAIPlatform, platform.sessions, platform.models, platform.data, local function calls

2.6 Summary

This summary consolidates the platform’s developer experience, workflow deployment, API communication, and SDK patterns into one cohesive architecture.

Where this is used later

Concepts explained in this chapter

← Chapter 1: Why your AI projects need a platform Chapter 3: The Model Service: Your platform’s gateway to AI models →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.