liveBook · Manning

Chapter 9: The Workflow Service: orchestrating and deploying AI applications

What you will learn in chapter 9 of Designing AI Systems

This introduction positions the Workflow Service as the platform component that turns developer workflow code into scalable HTTP applications by separating deployment operations from request-time execution.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction positions the Workflow Service as the platform component that turns developer workflow code into scalable HTTP applications by separating deployment operations from request-time execution.

Key terms: Workflow Service, workflow, platform services, gRPC contract, SDK client, control plane, data plane, workflow container, API Gateway, SDK

9.1 Workflow runtime server

This section introduces the SDK-embedded workflow runtime server that starts in each workflow container and exposes a decorated Python function as an HTTP service.

Key terms: workflow runtime server, SDK, @workflow decorator, HTTP service, workflow function, runtime server, GenAIPlatform class, service clients, container entry point

9.1.1 The @workflow decorator

This section explains how the @workflow decorator preserves a plain Python workflow function while attaching deployment and runtime metadata that the platform consumes at different lifecycle stages.

Key terms: @workflow, _workflow_metadata, deployment pipeline, runtime server, min_replicas, max_replicas, target_cpu_percent, target_memory_percent, cpu, memory

9.1.2 From decorated function to HTTP server

This section explains how the SDK runtime turns a decorated Python workflow into an independently deployed HTTP service and selects HTTP handling based on its response mode.

Key terms: runtime server, SDK, container entry point, @workflow, api_path, FastAPI, find_workflow, WORKFLOW_NAME, one-workflow-one-service, container image

9.1.3 Synchronous mode

Synchronous mode runs short workflows in a single open HTTP request while preserving tracing, validating inputs, enforcing end-to-end timeouts, and returning safe structured errors.

Key terms: synchronous mode, streaming mode, asynchronous mode, HTTP connection, distributed trace context, request headers, context variable, GenAIPlatform, platform.sessions, platform.data

9.1.4 Streaming mode

This section explains how streaming workflows progressively deliver model tokens as SSE events while safely detecting completion, timeouts, and errors.

Key terms: response_mode="stream", yield, platform.models.chat_stream(), iterator, tokens, Server-Sent Event, SSE, event_stream, data:, JSON

9.1.5 Asynchronous mode

This section explains how asynchronous workflow mode immediately returns a pollable job while the runtime executes long-running work in the background and the Workflow Service persists its state.

Key terms: response_mode="async", job record, job ID, background task, asyncio.create_task(), HTTP 202 Accepted, create_job(), processing, update_job_progress(), complete_job()

9.2 Async job lifecycle

This section introduces how asynchronous workflow jobs are stored, exposed through polling, and updated with developer-reported progress.

Key terms: async handler, platform.workflows.create_job(), job record, /jobs/{job_id}, progress reporting, polling clients, workflow executes

9.2.1 Job storage and the polling endpoint

This section defines the persistent job record and external polling endpoint that let asynchronous workflows report progress, return outcomes, and recover orphaned work.

Key terms: polling client, runtime server, crash recovery logic, job record, orphaned job, assigned endpoint, job_id, status, processing, completed

9.2.2 Progress reporting

This section explains how asynchronous workflows report progress, persist checkpoints for crash recovery, and record completion or failure through the Workflow Service.

Key terms: update_job_progress, _job_id_var, context variable, Workflow Service, /jobs/{job_id}, save_checkpoint, load_checkpoint, checkpoint_json, orphaned job, crash re-dispatch

9.3 Production readiness: concurrency, reliability, and health

This section introduces the production concerns the workflow runtime server must handle for AI applications: concurrent requests, retries for transient failures, and container health checks.

Key terms: runtime server, concurrency, automatic retry, transient network failures, health endpoints, container lifecycle, LLM providers, connection holding, event-loop starvation, rate limits

9.3.1 Concurrency inside a single container

This section explains how a workflow runtime handles many simultaneous requests within containers and scales safely across replicas under load.

Key terms: uvicorn, ASGI server, asyncio event loop, worker process, thread pool, gRPC, asyncio.to_thread, sync handler, streaming workflow, generator

9.3.2 Retry at the network layer

This section explains how SDK-level gRPC retries make workflow-to-platform-service calls resilient to transient network failures without changing workflow code.

Key terms: SDK retry, network layer, Model Service retry, workflow container, gateway, platform services, gRPC interceptor, outgoing call, UNAVAILABLE, DEADLINE_EXCEEDED

9.3.3 Health endpoints

This section explains how separate shallow liveness and readiness endpoints let the orchestrator manage workflow containers without turning transient dependency failures into outages.

Key terms: /health/live, liveness probe, /health/ready, readiness probe, container orchestrator, in-flight requests, graceful shutdown, termination signal, load balancing pool, HTTP 503

9.4 Workflow composition

This section introduces workflow composition as calling independently deployed workflows over their HTTP APIs instead of duplicating their logic.

Key terms: workflow composition, parent workflow, child workflows, API Gateway, HTTP APIs, duplication, composition, dependencies, scaling, team ownership

9.4.1 Calling other workflows

This section shows how a parent workflow invokes an independently deployed child workflow through the SDK and API gateway.

Key terms: platform.workflows.call(), api_path, request body, parent workflow, child workflow, HTTP POST, gateway, /knowledge/search, JSON response, container

9.4.2 Parallel workflow calls

This section explains how a parent workflow can call independent child workflows in parallel, the behavior and limits of that composition model, and when more formal orchestration is warranted.

Key terms: workflow.call_parallel(), workflow.call(), child workflow, parent workflow, gateway, retry logic, response mode handling, async, polls, exception propagates

9.4.3 Response mode handling

This section explains how SDK workflow calls hide child response modes, handle HTTP reliability, and when workflow composition is worth its network cost.

Key terms: platform.workflows.call(), sync response mode, async response mode, streaming response mode, text/event-stream, job ID, _http_post, HTTP retries, 5xx errors, 4xx errors

9.5 The Workflow Service contract

This section introduces the Workflow Service’s multi-consumer gRPC contract and organizes its operations into registry, deployment, and job groups.

Key terms: Workflow Service, gRPC contract, genai-platform deploy CLI, runtime server, external clients, SDK client, registry operations, deployment operations, job operations, workflow specification

9.5.1 Registry operations

This section defines how the Workflow Service registry records workflow specifications, supports their lifecycle operations, and supplies deployment infrastructure settings.

Key terms: registry, RegisterWorkflow, WorkflowSpec, api_path, response_mode, scaling configuration, ResourceConfig, cpu, memory, GPU

9.5.2 Deployment operations

This section defines how the Workflow Service deploys registered workflows, tracks their running state, routes traffic, rolls back versions, and formalizes async job operations.

Key terms: RegisterWorkflow, DeployWorkflow, WorkflowSpec, API Gateway, WorkflowDeployment, desired_replicas, current_replicas, healthy_endpoints, routing table, GetDeploymentStatus

9.6 Deployment pipeline

This section introduces the deployment pipeline details: how the CLI builds and publishes a workflow container and makes its API route available through the gateway.

Key terms: genai-platform deploy, container image, registry, Workflow Service, API gateway, routing table

9.6.1 What genai-platform deploy does

This section explains how genai-platform deploy packages a decorated workflow, registers and deploys it, and promotes one tested image across environments.

Key terms: genai-platform deploy, @workflow, _workflow_metadata, WorkflowSpec, RegisterWorkflow, DeployWorkflow, Dockerfile, WORKFLOW_MODULE, WORKFLOW_NAME, GENAI_GATEWAY_URL

9.6.2 Route registration with the API gateway

This section explains how the Workflow Service keeps the API Gateway’s routes mapped to the healthy container endpoints of each deployed workflow throughout its lifecycle.

Key terms: Workflow Service, API Gateway, routing table, api_path, WorkflowDeployment, readiness probes, auto-scaling, readiness probe, health check, drained replicas

9.7 Runtime management

This section explains how the Workflow Service translates workflow decorator settings into Kubernetes runtime resources that maintain availability, autoscale capacity, and safely roll out updates.

Key terms: Workflow Service, Kubernetes Deployment, HorizontalPodAutoscaler, @workflow decorator, min_replicas, resource parameters, CPU utilization, target_cpu_percent, max_replicas, stabilization windows

9.8 Summary

This summary connects the Workflow Service’s runtime, asynchronous execution, composition, and deployment control plane into one production architecture for AI workflows.

Where this is used later

Concepts explained in this chapter

← Chapter 8: The Experimentation Service: improving what your AI does Chapter 10: Building an AI assistant: putting the platform to work →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.