Chapter 2: Building with the platform: SDK and API design
This chapter introduces the SDK and API contracts that turn the platform’s services into productive developer and consumer interfaces.
After this chapter you will be able to
- SDK developer contract. Explain how the SDK contract lets application developers use and compose platform services while hiding infrastructure complexity. Describe that it supports orchestrating AI workflows and handling failures without requiring developers to manage the underlying platform details.
- Application API contract. Describe how the API contract exposes AI application functionality to frontend developers, mobile teams, and external integrators. Explain that the contract provides consistent interaction patterns, including synchronous, asynchronous, and streaming responses, along with concerns such as authentication.
- Developer-centered SDK design. Explain why the platform design starts from the developer experience rather than from infrastructure complexity. Describe the aim of envisioning an SDK interface that makes building AI applications feel effortless.
- Immediate value with minimal setup. Explain why the platform should let a developer start a working patient assistant with only a few lines of code and no upfront infrastructure or architectural decisions. Describe how this immediate productivity makes simple AI development approachable while leaving advanced customization available later.
- Configurable chat defaults. Describe the expectation that platform.models.chat() works out of the box with sensible defaults, while allowing developers to customize behavior when their needs become advanced. State that API keys, rate limiting, and response formats should not be prerequisites for the first successful use.
- Automatic conversation memory. Explain why a healthcare assistant needs conversation memory and why the platform should provide it automatically. Describe how memory lets the assistant interpret follow-up questions using prior conversational context rather than treating every request in isolation.
- Session Service memory operations. Describe the responsibilities of get_or_create(), get_messages(), and add_messages() in the proposed Session Service interface. Explain that the workflow retrieves or initializes a session, obtains ordered prior turns to assemble model context, and adds the interaction while the service manages context limits and persistence.
- Organizational knowledge integration. Explain why an AI application needs access to an organization's current documents rather than relying only on a model's training data or manually maintained prompts.
- Data Service semantic search. Describe the simple knowledge-search experience the Data Service should provide and the retrieval work it hides. Explain that a developer can upload or point to company documents and call platform.data.search(), while the platform processes documents, creates vector embeddings, performs semantic search, ranks relevant results, and formats them for the AI.
- Platform-enforced safety controls. Explain why healthcare AI safety must be enforced by platform technical controls rather than treated as an afterthought, left solely to model-provider safety, or manually reviewed conversation by conversation. Describe how this lets safety operate automatically once configured.
- Declarative healthcare safety policies. Describe how Sarah defines healthcare-specific policies, such as no_medical_advice and healthcare_compliance, once through the Guardrails Service rather than writing detection logic in every workflow. Explain that the platform applies those policies consistently while Sarah retains control over what safe behavior means in her context.
- Input and output validation. Distinguish input validation from output validation in the Guardrails Service. Describe how input validation checks what a patient asks and can gracefully redirect diagnosis or treatment requests, while output validation checks the AI response and blocks policy-violating medical advice before it reaches the patient.
- Declarative external tool registration. Describe how an external system is registered as a tool by declaring its location, credential reference, expected parameters, and supported operations. Explain that a developer registers the integration once rather than becoming an expert in each external API.
- Conversation tool coordination. Explain that, after tools are registered, the platform coordinates model calls and tool execution: it detects when a tool is needed, executes it, and incorporates the result into a meaningful response while maintaining conversation context. Describe the Tool Service as handling integration concerns such as authentication, parameter validation, error handling, and response formatting behind the scenes.
- Systematic prompt optimization. Explain how an evaluation setup enables systematic comparison and iterative improvement of prompt variations before an assistant is deployed. Describe the needed benchmark components: realistic test questions with expected answers, objective quality metrics, and comparison of prompts against the same benchmark.
- Multi-service workflow pipeline. Describe the patient-intake workflow as an ordered pipeline: validate input, retrieve session history, search knowledge, generate a tool-aware response, execute requested tools, validate output, and persist the exchange. Explain the role of the Session, Data, Model, Guardrails, and tool capabilities within that flow.
- Managed workflow infrastructure. Explain that the @workflow decorator turns business-logic code into a managed process, so the platform handles service coordination concerns such as retries, error recovery, state management, and observability. Describe automatic retry after a transient failure such as a knowledge-search timeout.
- Composable workflow design. Explain why smaller workflows with shared capabilities such as session retrieval, knowledge search, and safety validation should be composed into larger workflows instead of duplicating that coordination logic. State that the developer describes what should happen while the Workflow Service manages the orchestration details.
- Developer experience contract. Describe the platform's developer experience contract as immediate productivity, automatic memory, knowledge access, safety, tools, optimization, and orchestration delivered with minimal infrastructure burden on application developers.
- Service design constraints. Explain how the developer experience requirements become design constraints for the Model, Session, Data, Guardrails, Tool, and Workflow Services, including what each service abstracts or handles.
- Unified SDK abstraction. Explain why the SDK must present a unified interface across services so developers do not need to understand service boundaries or manually coordinate service interactions.
- Workflow deployment requirements. Describe the requirements a deployed @workflow Python function must meet: it must execute somewhere, access platform services such as the Model Service, and be reachable by a frontend application. Explain that deployment choices also determine whether teams can deploy independently and whether one team's bugs can affect another team's workflows.
- Shared process failure modes. Describe the operational, resource, and security problems caused by running multiple workflows in one shared process. Explain that one workflow's deployment, memory leak, infinite loop, or resource demand can disrupt unrelated workflows.
- Infrastructure-enforced workflow isolation. Explain why every workflow needs at least process-level isolation in its own environment. State that the infrastructure, not merely code organization, must enforce the boundary so unrelated workflows cannot interfere with one another.
- Workflow isolation tradeoffs. Distinguish plain processes, virtual machines, and containers by their isolation strength, resource overhead, and operational tradeoffs for running workflows.
- Container-based workflow boundaries. Explain how containers isolate each workflow as a self-contained packaged unit with its application dependencies, resource limits, and security boundaries.
- Workflow-to-service mapping. Explain the principle that one workflow equals one independently deployable service, including how a workflow defined with the @workflow decorator becomes its own service and deployment lifecycle.
- Independent workflow operations. Describe why grouping workflows creates release and resource coordination problems, and explain how independent services enable independent deployment, scaling, versioning, and per-workflow resource and cost attribution.
- Workflow operational registry. Describe the Workflow Service as the operational manager that maintains a registry of deployed workflows, including each workflow's container image, resource requirements, replica count, and health checks.
- Container lifecycle and scaling. Explain how the Workflow Service manages container lifecycle by creating initial containers, monitoring health, automatically replacing crashed instances, and adjusting replica counts from operational metrics such as CPU usage and traffic.
- Transparent platform connectivity. Explain that the Workflow Service ensures a workflow can reach platform services, such as the Session Service called through platform.sessions.get_or_create(), without the developer needing to know network topology or service locations.
- Workflow API contract. Explain why each deployable workflow needs a clear contract through which frontend applications, internal tools, mobile clients, and external partners can access it.
- Transport and interaction separation. Distinguish the transport mechanism clients use to communicate with workflows from the interaction patterns needed for AI operations that may take seconds, minutes, or stream results progressively.
- API sprawl definition. Explain API sprawl as the proliferation of inconsistent interfaces when teams independently expose their workflows. Describe how different URLs, authentication methods, error-handling approaches, and documentation make workflows feel like separate systems to consumers.
- Standardized workflow exposure. State that the platform must provide a consistent, standardized way to expose workflows to external clients in order to prevent API fragmentation. Identify the two required architectural decisions: how workflows map to accessible endpoints and which protocol they use to communicate.
- Workflow name and API path separation. Explain why a workflow’s internal name and its externally exposed API path are separate identifiers. Describe how an explicitly specified, stable API path lets developers refactor or rename a workflow without breaking external clients.
- Unified domain access structure. Describe how the platform exposes workflows beneath unified domains rather than giving each workflow an unrelated interface. Explain that domains can be organized around organizational or security boundaries, while workflows within one domain share authentication, response formats, and API conventions.
- HTTP workflow communication. Explain why HTTP is the protocol for accessing platform workflows across browsers, mobile apps, backend services, and command-line tools. Describe how standard methods, headers, and status codes make it a familiar, broadly supported interface rather than a custom protocol.
- AI timing interaction patterns. Describe why AI workflows cannot all use a simple request-response API. Distinguish quick operations, long-running operations, and conversational token-by-token streaming as timing needs that require three HTTP-based interaction patterns with different behaviors.
- AI interaction pattern selection. Distinguish synchronous, asynchronous, and streaming workflow interactions by the kind of AI operation they support. Explain that quick complete results use synchronous calls, long-running work uses asynchronous jobs, and progressively generated content uses streaming.
- Asynchronous job polling flow. Trace the asynchronous pattern for operations that take minutes or hours: a POST submits work and immediately returns job tracking information, while a separate GET status endpoint is polled for progress and eventually the completed result. Explain that this avoids fragile long-lived HTTP connections and lets work survive a client closing its browser or temporarily losing connectivity.
- Server-Sent Event streaming. Explain how streaming delivers AI-generated content progressively over Server-Sent Events: the client POSTs its input with an Accept: text/event-stream header, the server keeps the connection open, and sends token events as generation proceeds. Distinguish its goal of earlier visible output and better perceived responsiveness from asynchronous job polling.
- External and internal communication. Distinguish external client-to-workflow communication from internal workflow-to-platform-service communication, including the different environments in which they occur.
- Trust-boundary requirements. Explain why external and internal communication have fundamentally different requirements: external requests cross trust boundaries over the public internet, whereas internal calls occur among services the platform controls on its own network.
- API Gateway entry point. Explain why externally visible workflow URLs need an API Gateway between clients and containerized workflow replicas. Describe how the gateway gives clients a stable URL while workflows can deploy and scale independently.
- Centralized traffic reliability. Describe the operational responsibilities centralized in the API Gateway: credential validation, rate limiting, encrypted connections, health-based load balancing, and gradual deployment traffic shifting with rollback. Explain that centralizing them keeps this infrastructure logic out of individual workflow code.
- Internal protocol selection. Distinguish the communication priorities of external clients and internal platform services, and explain why the platform uses HTTP at the gateway but gRPC between services.
- gRPC remote-call abstraction. Describe how gRPC lets a workflow invoke a remote platform service through an SDK call that feels like a local function call, while generated client code performs the network and protocol work.
- End-to-end request flow. Trace Maria's request from the mobile app through the API gateway and workflow to the Session, Data, and Model Services, then back to the app. Describe how session context, retrieved company knowledge, and a model response are assembled before conversation state is updated.
- Gateway dual communication role. Explain the API gateway's distinct roles for external and internal traffic: a stable HTTPS boundary for applications and a gRPC proxy for workflow-to-service calls. State that it can route Protocol Buffer payloads without deserializing them.
- Gateway horizontal scaling. Explain how the API gateway acts as the single scaling point for platform traffic and scales horizontally. State that gateway instances are stateless, sit behind a load balancer, maintain their own service-registry view, and can route requests independently.
- Independent service scaling. Describe how individual platform services scale according to their own load profiles. Explain that the gateway service registry tracks replicas, distributes requests across healthy instances, and shields workflows from replica additions or removals.
- Zero-downtime traffic transition. Explain how gateway health checking supports zero-downtime deployments. Describe the sequence in which new containers register only after becoming healthy, receive traffic, and old containers complete in-flight requests before shutting down.
- SDK connective role. Explain how the SDK connects developer-written workflow code to distributed platform services, turning simple Python code into coordinated remote interactions.
- GenAIPlatform runtime entrypoint. Describe the GenAIPlatform object as the lightweight runtime entrypoint that gives workflow code access to platform services and anchors connection logic to the gateway.
- Gateway-mediated service communication. Explain how a workflow call such as platform.sessions.get_or_create reaches a platform service. Describe that the SDK creates a client that makes gRPC calls to the API gateway, which centrally routes them to the appropriate service container.
- Three-tier gateway configuration. Describe the three-tier resolution of the gateway URL in the GenAIPlatform constructor: an explicitly supplied URL takes priority, then GENAI_GATEWAY_URL, then localhost. Explain how this order lets the same workflow code run in development and production without code changes.
- Service client abstraction. Explain how the SDK lets workflow code call platform services such as Sessions, Models, and Data through local-looking Python methods while the clients hide the remote gRPC communication.
- Lazy client initialization. Describe the SDK's property-based lazy initialization: it stores empty client slots initially, creates a client only when an attribute such as platform.sessions is first accessed, and reuses that stored instance thereafter.
- Gateway metadata routing. Trace how a service client opens one secure gRPC channel to the stable API Gateway and attaches service-specific x-target-service metadata to every request so the gateway can route and load balance it to the intended healthy backend service.
- Workflow deployment metadata. Describe how the @workflow decorator captures deployment configuration by attaching a metadata dictionary to the wrapper function while preserving the original workflow function’s normal behavior.
- Deployment tool workflow. Trace the deployment tool’s sequence: import the Python file so decorators execute, discover functions carrying _workflow_metadata, build and push a container image, then register each workflow with the Workflow Service using its image location and deployment settings.
- Uniform SDK client pattern. Describe the common pattern used by every platform service client: a client in the SDK connects to the platform gateway, includes target-service routing metadata, and makes gRPC calls to the selected backend service.
- Infrastructure abstraction consistency. Justify the deliberate consistency of the SDK: service calls should feel like local function calls while the SDK handles gateway connection, the gateway handles routing, and backend services perform the actual work.
What it builds on
- Your AI platform blueprint: what we're building (chapter 1) · The platform blueprint and its collection of shared AI infrastructure services.
- Platform in action (chapter 1) · The platform-in-action view of applications using shared platform capabilities.
- Why platform thinking changes everything (chapter 1) · The argument that platform thinking provides value through shared, reusable infrastructure rather than isolated projects.
- Thinking from first principles: what AI applications actually need (chapter 1) · The identification of the core services AI applications need.
- The AI infrastructure reality: the model is just 2% of the story (chapter 1) · The distinction between the model itself and the broader infrastructure needed to operate an AI application.
Section by section
Introduction
This chapter introduces the SDK and API contracts that turn the platform’s services into productive developer and consumer interfaces.
- SDK developer contract: Explain how the SDK contract lets application developers use and compose platform services while hiding infrastructure complexity. Describe that it supports orchestrating AI workflows and handling failures without requiring developers to manage the underlying platform details.
Why it matters: The SDK is the layer that makes a collection of platform services practical for AI application developers. - Application API contract: Describe how the API contract exposes AI application functionality to frontend developers, mobile teams, and external integrators. Explain that the contract provides consistent interaction patterns, including synchronous, asynchronous, and streaming responses, along with concerns such as authentication.
Why it matters: A consistent API lets different kinds of clients consume AI applications without each application inventing its own interaction model. - Interface-driven platform design: Explain why the SDK and API contracts guide later platform architecture: service interfaces must match the ways developers build applications and the ways consumers use them. State that a platform without intuitive, powerful interfaces is only a collection of services rather than a source of developer productivity.
Why it matters: Designing interfaces first keeps later services aligned with real application and integration needs.
Key terms: SDK contract, platform services, AI workflows, infrastructure complexity, API contract, streaming responses, authentication, synchronous, asynchronous, developer productivity
2.1 Designing the ideal developer experience
This section frames the platform's SDK design around making AI application development effortless while leaving important behavior under developer control.
- Developer-centered SDK design: Explain why the platform design starts from the developer experience rather than from infrastructure complexity. Describe the aim of envisioning an SDK interface that makes building AI applications feel effortless.
Why it matters: Starting with what developers need helps the platform hide unnecessary infrastructure work instead of exposing it in every application. - Patient intake assistant scenario: Describe Sarah's patient intake assistant as the example used to define developer needs: it answers questions about appointments, insurance, and procedures; remembers conversations; accesses current company policies; and must not give medical advice. Explain that Sarah wants to focus on healthcare logic rather than infrastructure code.
Why it matters: The scenario makes concrete which AI capabilities should be simple for an application developer to use and which constraints must remain controllable. - Abstraction with control: Explain the desired balance in the developer experience: complex platform concerns should be invisible by default, while the important parts of the AI application remain controllable.
Why it matters: A useful platform reduces incidental complexity without taking away the developer's ability to shape critical application behavior.
Key terms: developer experience, SDK interface, infrastructure complexity, patient intake assistant, conversation memory, company policies, medical advice, healthcare logic, infrastructure code, complex parts invisible
2.1.1 Immediate productivity through effortless setup
This subsection establishes that the platform should let developers create a working AI assistant in a few lines of code by hiding infrastructure complexity behind sensible defaults.
- Immediate value with minimal setup: Explain why the platform should let a developer start a working patient assistant with only a few lines of code and no upfront infrastructure or architectural decisions. Describe how this immediate productivity makes simple AI development approachable while leaving advanced customization available later.
Why it matters: A platform earns adoption when developers can get useful results before they must understand its underlying complexity. - Model service complexity hiding: Describe the responsibilities the Model Service handles behind the simple chat interface: provider authentication, request formatting, response parsing, error handling, and cost tracking. Explain that these concerns are intentionally hidden from a developer getting started.
Why it matters: Hiding these operational details lets application developers focus first on their AI behavior rather than provider and infrastructure mechanics. - Configurable chat defaults: Describe the expectation that platform.models.chat() works out of the box with sensible defaults, while allowing developers to customize behavior when their needs become advanced. State that API keys, rate limiting, and response formats should not be prerequisites for the first successful use.
Why it matters: Sensible defaults reduce the barrier to first use without locking teams into a one-size-fits-all platform.
Key terms: immediate value, minimal setup, effortless simplicity, sensible defaults, Model Service, provider authentication, request formatting, response parsing, error handling, cost tracking
2.1.2 The conversation memory challenge
This section defines a developer-friendly Session Service interface that gives AI workflows automatic conversational memory without requiring developers to manage storage or context themselves.
- Automatic conversation memory: Explain why a healthcare assistant needs conversation memory and why the platform should provide it automatically. Describe how memory lets the assistant interpret follow-up questions using prior conversational context rather than treating every request in isolation.
Why it matters: Automatic memory makes multi-turn AI interactions feel natural while keeping infrastructure concerns out of application code. - Session Service memory operations: Describe the responsibilities of get_or_create(), get_messages(), and add_messages() in the proposed Session Service interface. Explain that the workflow retrieves or initializes a session, obtains ordered prior turns to assemble model context, and adds the interaction while the service manages context limits and persistence.
Why it matters: A small, clear interface lets workflows use reliable conversational state without embedding database and context-management logic.
Key terms: conversation memory, Session Service, session storage, conversation serialization, context window management, get_or_create(), get_messages(), add_messages(), sensible defaults, context window limits
2.1.3 The organizational knowledge problem
This section defines the need for a Data Service that lets AI applications use current organizational knowledge through simple, reliable semantic search.
- Organizational knowledge integration: Explain why an AI application needs access to an organization's current documents rather than relying only on a model's training data or manually maintained prompts.
Why it matters: Accurate answers about company-specific policies, procedures, and networks require knowledge that can change after a model was trained. - Data Service semantic search: Describe the simple knowledge-search experience the Data Service should provide and the retrieval work it hides. Explain that a developer can upload or point to company documents and call platform.data.search(), while the platform processes documents, creates vector embeddings, performs semantic search, ranks relevant results, and formats them for the AI.
Why it matters: The platform makes sophisticated retrieval infrastructure available without forcing every workflow developer to implement ingestion and search themselves. - Grounded current responses: Describe the expected outcome of knowledge integration: the AI returns accurate, current answers to organization-specific questions and cites the source documents that support them.
Why it matters: Grounding responses in current sources helps patients and developers trust answers about consequential organizational policies.
Key terms: organizational knowledge, model's training data, Data Service, knowledge integration, platform.data.search(), document processing, vector embeddings, semantic search, relevance ranking, accurate, current information
2.1.4 The safety imperative
This section establishes configurable, platform-enforced guardrails as the way to keep healthcare AI interactions safe without requiring every workflow to implement safety logic.
- Platform-enforced safety controls: Explain why healthcare AI safety must be enforced by platform technical controls rather than treated as an afterthought, left solely to model-provider safety, or manually reviewed conversation by conversation. Describe how this lets safety operate automatically once configured.
Why it matters: Platform-level enforcement makes critical safety behavior reliable and consistent across AI applications. - Declarative healthcare safety policies: Describe how Sarah defines healthcare-specific policies, such as no_medical_advice and healthcare_compliance, once through the Guardrails Service rather than writing detection logic in every workflow. Explain that the platform applies those policies consistently while Sarah retains control over what safe behavior means in her context.
Why it matters: Declaring policies once gives healthcare teams both consistent enforcement and control over their own safety requirements. - Input and output validation: Distinguish input validation from output validation in the Guardrails Service. Describe how input validation checks what a patient asks and can gracefully redirect diagnosis or treatment requests, while output validation checks the AI response and blocks policy-violating medical advice before it reaches the patient.
Why it matters: Checking both requests and responses protects patients while keeping unsafe interactions helpful rather than abruptly rejecting them.
Key terms: Guardrails Service, technical controls, platform capability, model safety, healthcare-specific policies, no_medical_advice, healthcare_compliance, add_policy, configuration, input validation
2.1.5 The external system integration challenge
This section defines a platform approach for integrating external systems: register tools declaratively and let the platform securely coordinate their use in model conversations.
- Declarative external tool registration: Describe how an external system is registered as a tool by declaring its location, credential reference, expected parameters, and supported operations. Explain that a developer registers the integration once rather than becoming an expert in each external API.
Why it matters: This lets application developers use clinic systems and other external capabilities without embedding integration details throughout conversational code. - Credential reference isolation: Explain that a tool registration names a credential rather than containing the actual API key, and that the platform resolves this name to the secret at execution time. State that this keeps the key out of code, repositories, and logs.
Why it matters: Referencing credentials by name prevents application code and conversational integrations from exposing external-system secrets. - Conversation tool coordination: Explain that, after tools are registered, the platform coordinates model calls and tool execution: it detects when a tool is needed, executes it, and incorporates the result into a meaningful response while maintaining conversation context. Describe the Tool Service as handling integration concerns such as authentication, parameter validation, error handling, and response formatting behind the scenes.
Why it matters: A conversational assistant can answer questions requiring live scheduling, eligibility, or other external data without the workflow manually orchestrating every API exchange.
Key terms: external tools, Tool Service, tool registration, parameters, operations, credential, API key, credential isolation, tool execution, model calls
2.1.6 The optimization dilemma
This section explains why prompt choices should be optimized through systematic evaluation rather than personal intuition.
- Systematic prompt optimization: Explain how an evaluation setup enables systematic comparison and iterative improvement of prompt variations before an assistant is deployed. Describe the needed benchmark components: realistic test questions with expected answers, objective quality metrics, and comparison of prompts against the same benchmark.
Why it matters: Systematic evaluation lets teams select prompts using measurable performance instead of subjective preference.
Key terms: prompt optimization, evaluation setup, test data, expected answers, metrics, benchmark
2.1.7 The Multi-Step Coordination Challenge
This section shows how a platform workflow coordinates AI services into a reliable, observable, and composable multi-step application pipeline.
- Multi-service workflow pipeline: Describe the patient-intake workflow as an ordered pipeline: validate input, retrieve session history, search knowledge, generate a tool-aware response, execute requested tools, validate output, and persist the exchange. Explain the role of the Session, Data, Model, Guardrails, and tool capabilities within that flow.
Why it matters: A production AI application needs its memory, knowledge, safety checks, model calls, and actions to work together rather than as isolated features. - Managed workflow infrastructure: Explain that the @workflow decorator turns business-logic code into a managed process, so the platform handles service coordination concerns such as retries, error recovery, state management, and observability. Describe automatic retry after a transient failure such as a knowledge-search timeout.
Why it matters: Managed infrastructure lets developers focus on what the application should do instead of repeatedly implementing production orchestration machinery. - Workflow observability: Describe the workflow's built-in observability: the platform automatically records step timing, failures, overall cost, quality scores, average duration, guardrail trigger rates, and detailed traces that locate and explain a pipeline failure.
Why it matters: End-to-end telemetry makes it possible to diagnose slow, failing, unsafe, or expensive AI workflows in production. - Composable workflow design: Explain why smaller workflows with shared capabilities such as session retrieval, knowledge search, and safety validation should be composed into larger workflows instead of duplicating that coordination logic. State that the developer describes what should happen while the Workflow Service manages the orchestration details.
Why it matters: Composition enables multiple AI applications, such as patient intake and insurance verification, to reuse reliable coordination patterns without copying infrastructure code.
Key terms: Session Service, Data Service, Model Service, Guardrails, registered tools, workflow, @workflow decorator, managed process, error recovery, automatic retries
2.1.8 Synthesis: the developer experience contract
This section synthesizes the developer experience contract: a unified platform should hide distributed AI infrastructure so developers can build sophisticated applications as naturally as business logic.
- Developer experience contract: Describe the platform's developer experience contract as immediate productivity, automatic memory, knowledge access, safety, tools, optimization, and orchestration delivered with minimal infrastructure burden on application developers.
Why it matters: This contract defines what the platform must make easy for every AI application developer. - Service design constraints: Explain how the developer experience requirements become design constraints for the Model, Session, Data, Guardrails, Tool, and Workflow Services, including what each service abstracts or handles.
Why it matters: Mapping user-facing promises to services keeps the platform architecture aligned with what developers need. - Unified SDK abstraction: Explain why the SDK must present a unified interface across services so developers do not need to understand service boundaries or manually coordinate service interactions.
Why it matters: A unified SDK is what turns separate distributed services into a coherent platform experience. - Developer experience north star: Justify using Sarah's ideal development experience as the north star for evaluating interface decisions, service boundaries, and implementation choices.
Why it matters: A shared evaluation standard prevents technical choices from drifting away from the platform's purpose.
Key terms: developer experience contract, immediate productivity, conversation memory, knowledge integration, safety controls, tool integration, Model Service, Session Service, Data Service, Guardrails Service
2.2 Code to container: the deployment story
This section frames the deployment questions and basic runtime requirements that arise after a developer defines a Python workflow with the @workflow decorator.
- Workflow deployment requirements: Describe the requirements a deployed @workflow Python function must meet: it must execute somewhere, access platform services such as the Model Service, and be reachable by a frontend application. Explain that deployment choices also determine whether teams can deploy independently and whether one team's bugs can affect another team's workflows.
Why it matters: These requirements drive the platform's choices for isolating, running, and exposing AI workflows.
Key terms: @workflow decorator, workflow, Model Service, frontend application, deployment
2.2.1 The shared process trap
This section shows why running all AI workflows in one shared process fails and establishes infrastructure-enforced, process-level isolation as the minimum requirement.
- Shared process failure modes: Describe the operational, resource, and security problems caused by running multiple workflows in one shared process. Explain that one workflow's deployment, memory leak, infinite loop, or resource demand can disrupt unrelated workflows.
Why it matters: Recognizing these failure modes explains why a multi-team AI platform cannot safely rely on one shared runtime. - Infrastructure-enforced workflow isolation: Explain why every workflow needs at least process-level isolation in its own environment. State that the infrastructure, not merely code organization, must enforce the boundary so unrelated workflows cannot interfere with one another.
Why it matters: Infrastructure-enforced isolation lets teams deploy, scale, and recover their workflows independently and safely. - Independent multi-team operation: Describe the independence that isolation provides to teams operating workflows: one team can deploy, consume more memory, or encounter a bug without requiring unrelated teams to coordinate or risking their production services.
Why it matters: Independent operation is what makes a shared platform practical for multiple teams and workloads. - Security boundary loss: Explain that sharing a process removes meaningful security boundaries between workflows, including workflows handling protected mental health information and public-facing content generators.
Why it matters: Sensitive workloads require enforced boundaries so that sharing a platform does not mean sharing exposure.
Key terms: shared process, workflow, memory leak, infinite loop, resource needs, process-level isolation, own environment, logical separation, infrastructure, workflow isolation
2.2.2 Choosing the right isolation mechanism
This section compares processes, virtual machines, and containers for workflow isolation and selects containers as the practical balance of security, resource efficiency, and operational needs.
- Workflow isolation tradeoffs: Distinguish plain processes, virtual machines, and containers by their isolation strength, resource overhead, and operational tradeoffs for running workflows.
Why it matters: Choosing an isolation model determines whether independent AI workflows can run safely and economically on the same platform. - Container-based workflow boundaries: Explain how containers isolate each workflow as a self-contained packaged unit with its application dependencies, resource limits, and security boundaries.
Why it matters: Per-workflow container boundaries prevent one AI application from disrupting another while making each deployment portable and consistent. - Container selection rationale: Justify choosing containers for the platform by connecting their seconds-level startup, kernel-based process isolation, resource limits, and service-specific network policies to the need to run many workflows economically and securely.
Why it matters: The platform needs an approach that protects workflows without the cost and delay of a full operating system per workflow.
Key terms: plain processes, virtual machines, VMs, containers, isolation, resource overhead, container, packaged environment, dependencies, process isolation
2.2.3 One workflow, one service
This section establishes that every workflow should be an independently deployable service to avoid coordination burdens and enable clean operational ownership.
- Workflow-to-service mapping: Explain the principle that one workflow equals one independently deployable service, including how a workflow defined with the @workflow decorator becomes its own service and deployment lifecycle.
Why it matters: This one-to-one boundary keeps a growing AI platform manageable by separating workflows operationally. - Independent workflow operations: Describe why grouping workflows creates release and resource coordination problems, and explain how independent services enable independent deployment, scaling, versioning, and per-workflow resource and cost attribution.
Why it matters: Independent operation lets teams change, scale, observe, and account for workflows without complex cross-workflow coordination.
Key terms: one workflow equals one independently deployable service, @workflow decorator, deployment lifecycle, one-to-one mapping, independent deployment, independent scaling, independent versioning, traffic shifting, resource attribution, cost tracking
2.2.4 Managing execution
This section explains how the Workflow Service centrally operates deployed workflow containers so developers can focus on business logic rather than production infrastructure.
- Workflow operational registry: Describe the Workflow Service as the operational manager that maintains a registry of deployed workflows, including each workflow's container image, resource requirements, replica count, and health checks.
Why it matters: A central registry gives the platform the information it needs to operate every workflow consistently. - Container lifecycle and scaling: Explain how the Workflow Service manages container lifecycle by creating initial containers, monitoring health, automatically replacing crashed instances, and adjusting replica counts from operational metrics such as CPU usage and traffic.
Why it matters: Automated lifecycle management and scaling keep workflows available and appropriately sized as demand changes. - Safe deployment traffic shifting: Describe the deployment approach in which the Workflow Service creates new containers while retaining old ones, gradually shifts traffic to the new version, and immediately reverts when problems appear.
Why it matters: Gradual traffic shifting and rollback reduce the risk of disrupting users with a faulty workflow release. - Transparent platform connectivity: Explain that the Workflow Service ensures a workflow can reach platform services, such as the Session Service called through platform.sessions.get_or_create(), without the developer needing to know network topology or service locations.
Why it matters: Hiding service-discovery and networking details lets developers write business logic while the platform supplies production connectivity.
Key terms: Workflow Service, registry, deployed workflows, container images, resource requirements, replica counts, lifecycle management, health monitoring, health checks, replicas
2.3 Exposing workflows as APIs
This section frames the need for a clear external API contract for deployed workflows, separating client communication transport from AI-specific interaction timing patterns.
- Workflow API contract: Explain why each deployable workflow needs a clear contract through which frontend applications, internal tools, mobile clients, and external partners can access it.
Why it matters: A consistent workflow interface lets different consumers integrate without bespoke access arrangements. - Transport and interaction separation: Distinguish the transport mechanism clients use to communicate with workflows from the interaction patterns needed for AI operations that may take seconds, minutes, or stream results progressively.
Why it matters: Separating communication foundation from timing behavior makes it possible to design APIs that fit both ordinary clients and AI workloads. - Integration consequences of interface choice: Describe how the chosen workflow interface affects ease of integration for frontend teams, mobile developers, and external partners, including whether they need special libraries or custom setup.
Why it matters: An interface is valuable only if the intended clients can adopt it with low integration friction.
Key terms: workflow, deployable service, external consumers, clear contract, transport mechanism, interaction patterns, stream results, AI operations, interface, frontend team
2.3.1 The API sprawl problem
This section defines API sprawl as inconsistent workflow interfaces and establishes standardized workflow exposure as the platform response.
- API sprawl definition: Explain API sprawl as the proliferation of inconsistent interfaces when teams independently expose their workflows. Describe how different URLs, authentication methods, error-handling approaches, and documentation make workflows feel like separate systems to consumers.
Why it matters: Naming API sprawl makes clear why shared platform infrastructure alone does not create a usable developer experience. - Consumer integration burden: Describe the practical cost of API sprawl for frontend developers, mobile teams, and external partners. Explain that they must maintain custom integration code and coordinate changes separately for every workflow, such as when an authentication scheme changes.
Why it matters: The consumer burden explains why interface consistency is a platform concern rather than a cosmetic API-design preference. - Standardized workflow exposure: State that the platform must provide a consistent, standardized way to expose workflows to external clients in order to prevent API fragmentation. Identify the two required architectural decisions: how workflows map to accessible endpoints and which protocol they use to communicate.
Why it matters: These two decisions turn shared workflow infrastructure into a coherent external platform interface.
Key terms: API sprawl, workflows, Workflow Service, custom integration code, authentication scheme, external clients, standardized way to expose workflows, accessible endpoints, communication protocol
2.3.2 Unified workflow exposure
This section defines how workflows receive stable developer-chosen API paths under unified domains so external access stays consistent and avoids API sprawl.
- Workflow name and API path separation: Explain why a workflow’s internal name and its externally exposed API path are separate identifiers. Describe how an explicitly specified, stable API path lets developers refactor or rename a workflow without breaking external clients.
Why it matters: Stable public paths protect client integrations while allowing the platform implementation to evolve. - Unified domain access structure: Describe how the platform exposes workflows beneath unified domains rather than giving each workflow an unrelated interface. Explain that domains can be organized around organizational or security boundaries, while workflows within one domain share authentication, response formats, and API conventions.
Why it matters: A shared domain-level access pattern prevents a collection of incompatible workflow APIs and supports appropriate security boundaries.
Key terms: workflow name, API path, api_path, public contract, external clients, unified domain structure, domain, logical boundary, security boundary, authentication mechanisms
2.3.3 The communication protocol
This section selects HTTP as the universal protocol for workflow access and explains why AI workflows require multiple HTTP interaction patterns for different execution times.
- HTTP workflow communication: Explain why HTTP is the protocol for accessing platform workflows across browsers, mobile apps, backend services, and command-line tools. Describe how standard methods, headers, and status codes make it a familiar, broadly supported interface rather than a custom protocol.
Why it matters: Using HTTP makes workflows reachable and testable from the tools and technology stacks developers already use. - AI timing interaction patterns: Describe why AI workflows cannot all use a simple request-response API. Distinguish quick operations, long-running operations, and conversational token-by-token streaming as timing needs that require three HTTP-based interaction patterns with different behaviors.
Why it matters: Matching the interaction pattern to workflow duration lets callers receive useful results without treating all AI work as an immediate web request.
Key terms: HTTP, HTTP methods, POST, GET, headers, status codes, request-response APIs, interaction patterns, streaming responses, token-by-token
2.3.4 Patterns of AI interaction
This section defines synchronous, asynchronous, and streaming HTTP interaction patterns for exposing AI workflows according to their duration and result-delivery needs.
- AI interaction pattern selection: Distinguish synchronous, asynchronous, and streaming workflow interactions by the kind of AI operation they support. Explain that quick complete results use synchronous calls, long-running work uses asynchronous jobs, and progressively generated content uses streaming.
Why it matters: Choosing the interaction pattern to match the work keeps an AI application responsive and reliable. - Synchronous HTTP response design: Describe the synchronous pattern for AI work that normally completes in under about 10 seconds: the client sends a POST, keeps the HTTP connection open briefly, and receives one complete JSON result. Explain why POST is used for submitted processing input and why JSON can return structured results or a URL for generated content stored elsewhere.
Why it matters: A conventional JSON request-response API is the simplest interface when users can reasonably wait for the complete AI result. - Asynchronous job polling flow: Trace the asynchronous pattern for operations that take minutes or hours: a POST submits work and immediately returns job tracking information, while a separate GET status endpoint is polled for progress and eventually the completed result. Explain that this avoids fragile long-lived HTTP connections and lets work survive a client closing its browser or temporarily losing connectivity.
Why it matters: Job submission and polling make long-running AI workflows usable without depending on one client connection remaining open. - Server-Sent Event streaming: Explain how streaming delivers AI-generated content progressively over Server-Sent Events: the client POSTs its input with an Accept: text/event-stream header, the server keeps the connection open, and sends token events as generation proceeds. Distinguish its goal of earlier visible output and better perceived responsiveness from asynchronous job polling.
Why it matters: Streaming lets users begin seeing and reading a generated response while the model is still producing it.
Key terms: synchronous requests, asynchronous requests, streaming responses, interaction patterns, POST, GET, JSON, object storage, synchronous interaction, job identifier
2.4 The communication architecture
This section frames the distinct external and internal communication problems that the platform must solve before tracing a request through the system.
- External and internal communication: Distinguish external client-to-workflow communication from internal workflow-to-platform-service communication, including the different environments in which they occur.
Why it matters: Treating these as separate communication problems lets the platform use an appropriate architecture for each. - Trust-boundary requirements: Explain why external and internal communication have fundamentally different requirements: external requests cross trust boundaries over the public internet, whereas internal calls occur among services the platform controls on its own network.
Why it matters: Trust boundaries and network context are why the platform should not assume one communication design fits every call. - End-to-end request tracing: Describe the request path the section proposes to examine: a patient asks Sarah's assistant a question, the request enters from an external client, reaches the appropriate workflow container, and then may invoke multiple internal platform services.
Why it matters: Following one request end to end makes the routing and internal-protocol decisions concrete.
Key terms: external communication, internal communication, workflows, platform services, trust boundaries, public internet, datacenter, network, Sarah's assistant, workflow container
2.4.1 The entry point problem
This section explains why an API Gateway is the platform's stable entry point and how it centralizes reliable request routing for independently deployed workflow containers.
- API Gateway entry point: Explain why externally visible workflow URLs need an API Gateway between clients and containerized workflow replicas. Describe how the gateway gives clients a stable URL while workflows can deploy and scale independently.
Why it matters: A single entry point lets application teams change and scale workflows without exposing container-level operational complexity to clients. - Centralized traffic reliability: Describe the operational responsibilities centralized in the API Gateway: credential validation, rate limiting, encrypted connections, health-based load balancing, and gradual deployment traffic shifting with rollback. Explain that centralizing them keeps this infrastructure logic out of individual workflow code.
Why it matters: Centralizing reliability and security controls avoids making every workflow team implement and coordinate the same operational machinery.
Key terms: API Gateway, single-entry point, workflow, containers, replicas, stable URL, authentication, rate limiting, encrypted connections, load balancing
2.4.2 The internal communication challenge
This section explains why platform services use gRPC internally while external clients communicate through HTTP, balancing universal access with efficient service-to-service calls.
- Internal protocol selection: Distinguish the communication priorities of external clients and internal platform services, and explain why the platform uses HTTP at the gateway but gRPC between services.
Why it matters: Choosing a protocol by communication context avoids making internal service calls pay the costs of a client-facing API. - gRPC remote-call abstraction: Describe how gRPC lets a workflow invoke a remote platform service through an SDK call that feels like a local function call, while generated client code performs the network and protocol work.
Why it matters: This abstraction lets workflow authors use platform services without manually building URLs, headers, request bodies, or network-handling code. - Protocol Buffers efficiency: Explain that gRPC uses the binary Protocol Buffers encoding rather than JSON, reducing payload size and serialization work so the benefit compounds when a workflow makes many internal calls per user request.
Why it matters: Small per-call savings become noticeable workflow performance improvements when requests fan out to multiple services.
Key terms: HTTP, gRPC, external clients, internal services, API gateway, SDK, Protocol Buffers, schema, generated client code, Session Service
2.4.3 Tracing the Complete Flow
This section traces one user request through the gateway, workflow, and platform services to show how centralized HTTPS and gRPC routing enables efficient, scalable, observable, and secure communication.
- End-to-end request flow: Trace Maria's request from the mobile app through the API gateway and workflow to the Session, Data, and Model Services, then back to the app. Describe how session context, retrieved company knowledge, and a model response are assembled before conversation state is updated.
Why it matters: A complete request trace makes clear how an AI workflow combines multiple platform capabilities into one user-facing response. - Gateway dual communication role: Explain the API gateway's distinct roles for external and internal traffic: a stable HTTPS boundary for applications and a gRPC proxy for workflow-to-service calls. State that it can route Protocol Buffer payloads without deserializing them.
Why it matters: Using one gateway for both boundaries centralizes routing while preserving the efficiency of internal gRPC communication. - Centralized gateway operations: Describe how the gateway centralizes service discovery, observability, and authorization policy for internal calls. Explain that workflows call the gateway rather than tracking service replicas, and the gateway can trace, meter, and authorize a request consistently.
Why it matters: Centralizing these cross-cutting concerns lets services scale and evolve without forcing every workflow or service to implement them separately.
Key terms: API gateway, workflow, Session Service, Data Service, Model Service, HTTPS, gRPC, Protocol Buffer, TLS termination, gRPC proxy
2.4.4 Why these decisions scale
This section explains how the gateway-based communication architecture scales traffic, services, deployments, and internal gRPC calls without forcing workflow developers to manage distributed-systems complexity.
- Gateway horizontal scaling: Explain how the API gateway acts as the single scaling point for platform traffic and scales horizontally. State that gateway instances are stateless, sit behind a load balancer, maintain their own service-registry view, and can route requests independently.
Why it matters: A stateless gateway tier lets the whole platform absorb increasing external and internal traffic without redesigning workflows. - Independent service scaling: Describe how individual platform services scale according to their own load profiles. Explain that the gateway service registry tracks replicas, distributes requests across healthy instances, and shields workflows from replica additions or removals.
Why it matters: Services such as sessions, models, and data can receive the capacity they need without over-scaling unrelated parts of the platform. - Zero-downtime traffic transition: Explain how gateway health checking supports zero-downtime deployments. Describe the sequence in which new containers register only after becoming healthy, receive traffic, and old containers complete in-flight requests before shutting down.
Why it matters: This deployment behavior allows teams to update many workflows and services frequently without dropped requests or maintenance windows. - gRPC proxy efficiency: Explain why gateway proxying preserves gRPC efficiency. State that the gateway forwards binary Protocol Buffer data without translation or deserialization, so internal calls retain compact encoding and efficient serialization even with a gateway hop.
Why it matters: Preserving efficient internal calls prevents the cost of repeated service-to-service communication from becoming large at high request rates.
Key terms: API gateway, horizontal scaling, load balancer, stateless, service registry, independent scaling, load profiles, replicas, healthy instances, health checking
2.5 Building the SDK
This section introduces the SDK as the bridge that turns developer-written Python workflows into coordinated calls to distributed platform services and deployable workflow metadata.
- SDK connective role: Explain how the SDK connects developer-written workflow code to distributed platform services, turning simple Python code into coordinated remote interactions.
Why it matters: The SDK hides distributed-systems plumbing so developers can write workflows through a simple programming interface. - GenAIPlatform runtime entrypoint: Describe the GenAIPlatform object as the lightweight runtime entrypoint that gives workflow code access to platform services and anchors connection logic to the gateway.
Why it matters: A single platform object gives developers a consistent way to locate and use platform capabilities from workflow code. - Workflow SDK infrastructure responsibilities: State the SDK responsibilities implied by the simple workflow: creating an object that can find and connect to services, invoking remote service methods through gRPC, and capturing deployment information through the decorator.
Why it matters: Recognizing the hidden responsibilities behind concise workflow code explains why the SDK must be designed as more than a collection of convenience methods.
Key terms: SDK, workflow code, distributed platform services, gRPC, GenAIPlatform object, runtime entrypoint, gateway connection logic, service-specific clients, decorator, deployment information
2.5.1 Connection to the API Gateway
This section explains how the SDK platform object connects isolated workflow containers to platform services through an API gateway, using environment-aware gateway configuration.
- Gateway-mediated service communication: Explain how a workflow call such as platform.sessions.get_or_create reaches a platform service. Describe that the SDK creates a client that makes gRPC calls to the API gateway, which centrally routes them to the appropriate service container.
Why it matters: This lets workflow code use platform services even though the workflow and those services run as separate networked processes. - Three-tier gateway configuration: Describe the three-tier resolution of the gateway URL in the GenAIPlatform constructor: an explicitly supplied URL takes priority, then GENAI_GATEWAY_URL, then localhost. Explain how this order lets the same workflow code run in development and production without code changes.
Why it matters: The configuration fallback makes service connectivity flexible while keeping the common development and deployment paths effortless.
Key terms: API gateway, gRPC, Session Service, service containers, client object, GenAIPlatform, gateway URL, GENAI_GATEWAY_URL, localhost, constructor
2.5.2 Providing access to services
This section shows how the SDK exposes distributed platform services as local-looking clients that are lazily created and routed through a central gRPC API Gateway.
- Service client abstraction: Explain how the SDK lets workflow code call platform services such as Sessions, Models, and Data through local-looking Python methods while the clients hide the remote gRPC communication.
Why it matters: This gives workflow authors a simple programming interface without requiring them to manage distributed-service communication directly. - Lazy client initialization: Describe the SDK's property-based lazy initialization: it stores empty client slots initially, creates a client only when an attribute such as platform.sessions is first accessed, and reuses that stored instance thereafter.
Why it matters: Lazy initialization avoids opening channels and allocating clients for services a workflow never uses. - Gateway metadata routing: Trace how a service client opens one secure gRPC channel to the stable API Gateway and attaches service-specific x-target-service metadata to every request so the gateway can route and load balance it to the intended healthy backend service.
Why it matters: Centralized gateway routing means clients do not need to track changing individual service instances as services scale or restart.
Key terms: GenAIPlatform, service client, SessionClient, ModelClient, DataClient, gRPC, lazy initialization, @property, _sessions, API Gateway
2.5.3 Capturing deployment information
This section shows how the SDK’s @workflow decorator attaches deployment metadata to a Python function and how the deployment tool uses that metadata to build, publish, and register a workflow.
- Workflow deployment metadata: Describe how the @workflow decorator captures deployment configuration by attaching a metadata dictionary to the wrapper function while preserving the original workflow function’s normal behavior.
Why it matters: This lets the same workflow function run locally for testing while carrying the information the platform needs to deploy it. - Deployment configuration fields: State the deployment settings captured by the decorator and explain what each controls: unique workflow name, external API path, replica counts and CPU scaling threshold, and synchronous, asynchronous, or streaming response mode.
Why it matters: These settings connect a workflow’s code to its public endpoint, scaling behavior, and client interaction pattern. - Deployment tool workflow: Trace the deployment tool’s sequence: import the Python file so decorators execute, discover functions carrying _workflow_metadata, build and push a container image, then register each workflow with the Workflow Service using its image location and deployment settings.
Why it matters: Understanding this sequence explains how Python source code becomes a running, registered platform workflow.
Key terms: @workflow decorator, wrapper function, _workflow_metadata, deployment configuration, workflow name, API path, replica counts, CPU threshold, auto-scaling, response mode
2.5.4 The SDK foundation
This section establishes the SDK pattern that gives developers uniform access to platform services and hides distributed infrastructure behind simple client calls and deployment commands.
- Uniform SDK client pattern: Describe the common pattern used by every platform service client: a client in the SDK connects to the platform gateway, includes target-service routing metadata, and makes gRPC calls to the selected backend service.
Why it matters: Learning one client pattern lets developers use every platform service without learning a different connection and routing mechanism each time. - Platform object service access: Explain how creating GenAIPlatform() gives a developer a gateway-aware platform object whose service properties, such as platform.sessions, provide ready-to-use clients.
Why it matters: A single platform object gives application code a consistent starting point for accessing platform capabilities. - Infrastructure abstraction consistency: Justify the deliberate consistency of the SDK: service calls should feel like local function calls while the SDK handles gateway connection, the gateway handles routing, and backend services perform the actual work.
Why it matters: This separation lets developers focus on application workflows instead of distributed-system connection and routing details. - Workflow deployment integration: State that decorating a function with @workflow provides deployment information that lets the deployment tool package and deploy that workflow.
Why it matters: The workflow decorator connects application code to a deployment process designed to feel like a single command.
Key terms: *Client, platform gateway, x-target-service, gRPC calls, routing metadata, GenAIPlatform, platform.sessions, platform.models, platform.data, local function calls
2.6 Summary
This summary consolidates the platform’s developer experience, workflow deployment, API communication, and SDK patterns into one cohesive architecture.
Where this is used later
- Integrating with the SDK (chapter 3) · The ModelClient integrates the Model Service into the SDK interface.
- Integrating with the SDK (chapter 4) · The Session Service is made available through SDK integration.
- Workflow runtime server (chapter 9) · Workflow authoring uses the developer-facing programming interface introduced by this contract.
- The model service contract (chapter 3) · The Model Service contract must fit the developer-facing ways of invoking AI providers.
- The Workflow Service contract (chapter 9) · The Workflow Service contract formalizes capabilities that must align with application orchestration needs.
- The simplest assistant: a model call in a workflow (chapter 10) · The simplest assistant begins with a model call in a workflow.
- Observability: Cost tracking and metrics (chapter 3) · Cost tracking becomes part of the Model Service's observability responsibilities.
- Rate limiting (chapter 3) · Rate limiting becomes a Model Service concern that developers need not manage to get started.
Concepts explained in this chapter
- HTTPHTTP is the stateless application-layer protocol used to transfer requests and responses between networked clients and services.
- workflowA workflow is a defined sequence of dependent operations that coordinates processing steps and manages their execution state.
- stream processingA streaming response is an output delivery mode that sends generated content incrementally as it becomes available instead of waiting for the complete result.
- API gatewayAn API gateway is a gateway component that provides a unified entry point for accessing backend services and applies shared cross-cutting controls.
- asynchronous requestAn asynchronous request is an interaction in which the client submits work, receives an acknowledgment or job identifier, and retrieves progress or results later.
- server-sent eventsServer-Sent Events is an HTTP-based mechanism for continuously delivering unidirectional, incrementally generated events from a server to a client.
- synchronous callSynchronous execution is an execution model in which the caller waits for an operation to complete before receiving its result.
- software development kitA software development kit is a collection of programming tools, libraries, documentation, and related resources for building applications against a software platform or service.
- gRPCgRPC is a high-performance remote procedure call framework that uses service definitions and serialized messages for communication between software components.
- developer experienceDeveloper experience is the quality and usability of the workflows, interfaces, and tools that developers use to build and operate software.
- protobufProtocol Buffers is a language-neutral, platform-neutral serialization format for structured data and service interface definitions.
- lazy initializationLazy initialization is the creation of an object or resource only when it is first needed.
- workflow serviceA workflow service is a platform service that coordinates sequential and parallel operations, state management, retries, and failure recovery in complex processes.
- routing metadataRouting metadata is request-associated key-value information that identifies the backend service to which a gateway should direct a call.
- retryA retry is a repeated attempt to perform an operation after a transient failure or unsuccessful attempt.
- interfaceA consistent interface is a uniform software interface that presents the same interaction model across different implementations or providers.
- workflow decoratorA workflow decorator is a function annotation that converts an ordinary function into a managed workflow process.
- multi-step orchestrationMulti-step orchestration is the coordination of a sequence of dependent AI workflow steps with step-specific capabilities, error handling, and recovery.
- service discoveryService discovery is the mechanism by which distributed components locate available services and their connection endpoints.
- api sprawlAPI sprawl is the uncontrolled proliferation of separate API interfaces with fragmented conventions and access patterns.
- independent deploymentAn independently deployable service is a service that can be released, scaled, and versioned without coordinating deployment with other services.
- guardrailA guardrail is an execution policy that constrains AI system inputs, actions, and outputs according to safety, business, permission, or regulatory requirements.
- data serviceA data service is a reusable platform service that ingests documents, generates vector embeddings, and supports semantic search over organizational knowledge.
- tool serviceA platform service that registers and discovers external tools, APIs, and services while handling authentication, rate limiting, recovery, and caching.
- tool usetool_calls is a message field containing an assistant's requested tool or function invocations.
- memoryModel-managed memory is a capability in which an AI model selects, stores, and retrieves information from persistent memory outside its immediate context window.
- session serviceA session service is a platform service that stores and manages conversational state, user preferences, interactions, and conversation history across sessions.
- contractAn API is a defined interface through which software components communicate and invoke capabilities, including interfaces exposed by external systems.
- model serviceA model service is a platform service that provides a unified interface to multiple AI model providers while handling provider-specific APIs, routing, resilience, and operational concerns.
- autoscalingAuto-scaling is the automatic adjustment of computing resources in response to workload demand.
- load balancingLoad-based routing distributes requests among providers or execution paths according to their current workload or capacity.
- evaluationEvaluation is the systematic assessment of an AI system or change against defined evidence and criteria.
- platformA software and infrastructure foundation that provides shared capabilities for building and operating applications.
- platform serviceA reusable infrastructure or software service provided centrally to support multiple applications or teams.
- observabilityThe ability to infer a system's internal behavior and health from its emitted telemetry and operational records.
- recoveryCrash recovery is the detection and re-dispatch of work whose executing process or container has unexpectedly disappeared.
- service coordinationService coordination is the management of dependencies and interactions among components of a service-based system.
- container imageA container image is an immutable package containing application code, runtime components, libraries, and dependencies for creating containers.
- deployment modelDeployment configuration is the set of settings that specifies how a workflow is exposed, scaled, and run on a platform.
- safety validationSafety validation is the systematic checking of inputs, outputs, and intermediate operations against safety, privacy, and policy requirements.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
