Chapter 9: The Workflow Service: orchestrating and deploying AI applications
This introduction positions the Workflow Service as the platform component that turns developer workflow code into scalable HTTP applications by separating deployment operations from request-time execution.
After this chapter you will be able to
- Workflow Service purpose. Explain why the Workflow Service is needed in addition to the platform services for models, sessions, data, tools, guardrails, and observability. State that it turns developer workflow code into a running, scalable application that external clients can call.
- Control and data planes. Distinguish the Workflow Service control plane from its data plane. Describe the control plane as the platform service that deploys and operates workflows, and the data plane as SDK code inside each workflow container that handles each HTTP request and invokes the developer's function.
- Workflow runtime server role. Explain that the workflow runtime server listens for HTTP requests, invokes the developer's workflow function, and returns responses, thereby turning a decorated Python function into a production HTTP service.
- Decorator metadata model. Explain how @workflow attaches a _workflow_metadata dictionary to a workflow function without changing the function's normal calling behavior. Describe the deployment pipeline and runtime server as separate consumers of that same metadata at deployment time and container startup.
- Scaling and resource configuration. Describe the scaling and resource metadata declared by @workflow, including replica bounds, CPU or memory utilization thresholds, per-container CPU and memory allocation, and optional GPU requests. Explain when memory-based scaling is a better saturation signal than CPU-based scaling.
- Reliability metadata enforcement. Explain how timeout_seconds limits one request and returns a 504 when the runtime server terminates an overlong execution, and how max_retries configures automatic exponential-backoff retries for transient gRPC platform-service failures. Distinguish the retry choices appropriate for a tight synchronous workflow from those suitable for a long-running asynchronous workflow.
- Runtime server transformation. Explain how the workflow runtime server turns a decorated Python function into an HTTP endpoint when its container starts. Describe that the runtime is SDK library code used as the container entry point, rather than a separate service.
- Workflow container selection. Describe how the runtime server uses the WORKFLOW_NAME environment variable to identify the workflow that a container serves. Explain why multiple decorated functions in one Python file still become separate container images and independently scalable, deployable, and versioned services.
- Response mode translation. Distinguish how the runtime server translates synchronous, streaming, and asynchronous workflow outputs into HTTP responses. State that metadata response_mode selects a different handler because each interaction pattern has different HTTP response mechanics.
- Synchronous mode suitability. Describe when synchronous mode is appropriate and when streaming or asynchronous mode should replace it. State that the HTTP connection remains open while the workflow runs.
- Automatic trace-context propagation. Explain how the synchronous handler extracts distributed trace context from request headers, places it in a context variable, and lets a GenAIPlatform instance automatically carry that context into session, data, and model calls.
- Request validation and safe errors. Describe how the handler validates JSON request arguments against the workflow function signature and how it maps invalid input and unhandled exceptions to safe HTTP JSON errors.
- End-to-end timeout handling. Explain how the runtime applies the decorator's timeout_seconds to the whole workflow with asyncio.wait_for, runs synchronous code in a separate thread, and propagates the remaining time budget as downstream gRPC deadlines.
- Streaming workflow pattern. Describe how a workflow is written and configured for streaming: it sets response_mode="stream", calls platform.models.chat_stream(), iterates over progressive tokens, and yields each value rather than returning one complete response.
- SSE stream handling. Explain how the runtime server turns a streaming workflow generator into Server-Sent Events: it validates and traces the request, serializes each yielded dictionary as JSON, and emits it with the SSE data: prefix and double-newline event boundary.
- Chunk timeout and completion. Explain why the streaming handler obtains generator chunks with next(gen, _DONE) inside asyncio.wait_for: the timeout detects a stalled model service per chunk, while the _DONE marker signals normal generator exhaustion without a StopIteration error in a task. Describe that mid-stream failures are sent as structured error events before the stream closes.
- Asynchronous execution flow. Trace what happens when a workflow uses response_mode="async", from the incoming HTTP request through job creation, background execution, and the immediate HTTP 202 response.
- Job status and failure reporting. Describe how the asynchronous handler and background runner record processing, progress, completion, and failure so polling clients can observe a job's outcome.
- Runtime and service responsibilities. Distinguish the workflow container's responsibility for executing developer code from the Workflow Service's responsibility for owning persistent job state and serving the polling URL.
- Async job lifecycle questions. State the two unresolved aspects of asynchronous execution: where the job record created by platform.workflows.create_job() lives and contains, and how the /jobs/{job_id} polling endpoint serves status before completion.
- Polling progress reporting. Describe the planned mechanism by which workflow developers report progress while a workflow executes, so polling clients can receive progress information between submission and completion.
- Job record consumer requirements. Explain how the polling client, runtime server, and crash recovery logic each use the same job record, and identify the information each consumer needs.
- Universal jobs table design. Describe the jobs table fields and access patterns that support asynchronous workflow execution, including lifecycle status, progress, generic JSON payloads, and recovery queries.
- Polling endpoint lifecycle. Trace the asynchronous job lifecycle from job creation through the 202 response and repeated GET requests to /jobs/{job_id}, including how polling responses differ for processing, completed, and failed jobs.
- Context-based progress reporting. Explain how an asynchronous workflow updates a job's progress without the developer supplying a job ID. Describe how the SDK obtains the job ID from a context variable and how a later poll returns the updated progress message.
- Checkpoint-based crash recovery. Describe how save_checkpoint and load_checkpoint let an asynchronous workflow resume after a crash. Explain that saved intermediate results are JSON-serialized into the job record, then returned when an orphaned job is re-dispatched to a new container with the same input.
- Production runtime concerns. Describe the three production concerns handled by the workflow runtime beyond basic request processing: concurrency for simultaneous requests, automatic retry for transient network failures, and health endpoints for container lifecycle management.
- AI-specific production pressure. Explain why AI workflows make standard concurrency and reliability patterns especially important: LLM calls can hold connections for seconds or minutes, and third-party providers can impose rate limits, capacity constraints, and transient timeouts.
- Event-loop and worker concurrency. Explain how Uvicorn's asyncio event loop multiplexes requests that are waiting on platform-service gRPC calls, and how multiple worker processes increase parallel request handling inside one container.
- Execution modes under load. Describe how synchronous, streaming, and asynchronous workflows are dispatched so that the runtime can continue handling other requests.
- Request state isolation. Explain why simultaneous requests in the same container do not contaminate each other's platform clients, retry state, trace context, or job IDs.
- SDK retry scope. Distinguish SDK network-layer retries from the Model Service retries used for external model providers. Explain that SDK retries cover calls from a workflow container through the gateway to platform services, while Model Service retries cover calls from the Model Service to providers such as OpenAI or Anthropic.
- Interceptor retry behavior. Describe how a gRPC retry interceptor wraps outgoing SDK calls, returns successful responses normally, and retries only designated transient failures. State that UNAVAILABLE, DEADLINE_EXCEEDED, and RESOURCE_EXHAUSTED are retryable because they represent temporary unreachability, timeout, or overload, whereas invalid requests propagate immediately.
- Backoff and inherited configuration. Explain that retry attempts use exponential backoff of 100, 200, then 400 milliseconds to avoid hammering a struggling service, and that BaseClient attaches the interceptor to its gRPC channel with grpc.intercept_channel. Describe how a workflow's decorator metadata supplies max_retries once, causing all SDK service clients in that workflow to inherit that retry limit.
- Liveness and readiness roles. Distinguish the /health/live liveness probe from the /health/ready readiness probe, including what a successful HTTP response means for each and how the orchestrator acts on failure.
- Startup and shutdown traffic control. Describe how readiness controls traffic during startup and graceful shutdown: it stays unavailable while initialization occurs, then returns 503 on termination so in-flight requests can finish before the container exits.
- Shallow health-check design. Justify keeping liveness and readiness probes shallow by checking only that the server process is responsive, while handling temporary dependency connectivity failures with per-request SDK retry logic.
- Workflow composition model. Explain how a parent workflow can compose capabilities by calling other deployed workflows through the API Gateway, using the same HTTP APIs available to external clients.
- Composition over duplication benefits. Justify choosing workflow composition over copying search and summarization logic by describing independent updates, isolated dependencies and resource requirements, scaling, and team ownership.
- Child workflow invocation. Describe how a parent workflow uses platform.workflows.call() with a child workflow api_path and request body to invoke that workflow and receive its result as a dictionary.
- Gateway-routed workflow execution. Trace the path of a child-workflow call: the SDK sends an HTTP POST through the gateway to the child's api_path, the gateway routes it to a container running that workflow, and the parent continues after the JSON response returns.
- Parallel child workflow calls. Explain when to use workflow.call_parallel() instead of sequential workflow.call() calls and describe how it runs independent child workflows concurrently. State that the SDK waits for all calls and returns their results in the same order as the input list.
- Parallel failure and response handling. Describe how call_parallel() handles child failures and response modes: each call uses the normal call() path with its retries and response handling, including polling async children; a child failure after retries propagates an exception to the parent immediately.
- Code-defined composition tradeoffs. Distinguish code-sequenced composition with call() and call_parallel() from a dedicated workflow orchestrator. Explain that ordinary Python makes dependency flow implicit and readable, but the platform cannot see the graph to retry individual steps, checkpoint between calls, or show step-level progress; complex shared-dependency or expensive-step pipelines may need Argo or Metaflow.
- Transparent child response handling. Explain how platform.workflows.call() inspects a child workflow's HTTP response mode and hides sync, async, and streaming differences from the parent developer. Describe the return path for a 200 JSON sync response, a 202 async response, and a text/event-stream response.
- Composition boundary decisions. Justify choosing workflow composition for real boundaries such as shared cross-team logic, dependency isolation, or independent scaling, rather than composing simple logic with shared dependencies under one team's ownership. State the network-hop and gateway-round-trip cost of each child call.
- Multi-consumer contract design. Explain why the Workflow Service contract differs from other platform service contracts by serving three consumers rather than only workflow code using an SDK client. Identify the deploy CLI, the runtime server in workflow containers, and external clients as those consumers.
- Three operation groups. Describe how the Workflow Service contract maps its three consumers' needs into registry, deployment, and job operations. State the major responsibilities covered by each group.
- Workflow registration specification. Describe how a workflow enters the platform through RegisterWorkflow and identify the key contents of its WorkflowSpec.
- Resource and GPU configuration. Explain how ResourceConfig and GPU fields translate a workflow's resource requirements into container limits and Kubernetes node scheduling.
- Deployment realization process. Explain how DeployWorkflow turns a registered WorkflowSpec into running containers and API Gateway routes, unlike RegisterWorkflow, which records the intended workflow without starting it.
- Deployment state and routing. Describe how a WorkflowDeployment represents desired and actual replica state, and how healthy_endpoints drives API Gateway routing as replicas are created or removed.
- Deployment status and rollback. Trace deployment status from deploying to healthy or failed, and explain that RollbackWorkflow redeploys an older registered WorkflowSpec using that version's container image.
- Deployment pipeline overview. Describe the end-to-end deployment flow introduced here: a developer runs genai-platform deploy, which builds a container image, pushes it to a registry, registers the workflow with the Workflow Service, and results in the workflow API path being routed by the gateway.
- Six-step deployment pipeline. Trace the six steps performed by genai-platform deploy: discover decorated workflows, read their metadata, generate or use a Dockerfile, build and push an image, register a WorkflowSpec, and trigger deployment while polling for its result.
- Container runtime contract. Describe how the generated Dockerfile uses cached layers and environment variables to start the SDK runtime server for a particular workflow. State the minimum custom-Dockerfile requirements: install genai-platform-sdk, set WORKFLOW_NAME, and use the runtime server as the entrypoint.
- Build-once promotion discipline. Justify build-once, promote-many deployment: build an image once, then use promote to redeploy that identical version to later environments without rebuilding, while allowing environment-specific targets, gateway endpoints, and scaling settings.
- Gateway route registration. Explain how the Workflow Service registers a deployed workflow with the API Gateway after its container replicas pass readiness probes. State that it sends the workflow’s api_path and its healthy endpoint addresses so gateway requests can be forwarded to that workflow.
- Dynamic route lifecycle. Describe how the Workflow Service updates the API Gateway routing table as replicas are added, removed, replaced, or deleted. Explain that deleting a workflow removes its route entirely, so requests to its former API path receive a 404 response.
- Kubernetes runtime resources. Describe the two Kubernetes resources the Workflow Service generates for each workflow and what the Kubernetes Deployment is responsible for at runtime.
- CPU-based autoscaling behavior. Explain how the HorizontalPodAutoscaler changes workflow replica counts from CPU utilization, including the min/max bounds and asymmetric scale-up and scale-down stabilization windows.
- Metric-driven rollout rollback. Trace how an updated workflow is rolled out and rolled back: Kubernetes performs a rolling pod replacement, while the Workflow Service compares application error rates with the prior version baseline to decide whether to revert.
What it builds on
- Code to container: the deployment story (chapter 2) · The code-to-container deployment story and the one-workflow-one-service design.
- Exposing workflows as APIs (chapter 2) · The design for exposing workflows as HTTP APIs.
- Building the SDK (chapter 2) · The SDK foundation that captures workflow deployment information and reaches platform services.
- The communication architecture (chapter 2) · The external HTTP entry point, API Gateway routing, and internal gRPC communication architecture.
- One workflow, one service (chapter 2) · The one-workflow-one-service approach for isolating and deploying workflows.
- Unified workflow exposure (chapter 2) · Unified HTTP exposure for workflows.
- The entry point problem (chapter 2) · The API Gateway as the external entry point for workflow requests.
- The SDK foundation (chapter 2) · The SDK foundation that supplies platform functionality to developer code.
Section by section
Introduction
This introduction positions the Workflow Service as the platform component that turns developer workflow code into scalable HTTP applications by separating deployment operations from request-time execution.
- Workflow Service purpose: Explain why the Workflow Service is needed in addition to the platform services for models, sessions, data, tools, guardrails, and observability. State that it turns developer workflow code into a running, scalable application that external clients can call.
Why it matters: The Workflow Service is the layer that makes individually useful AI capabilities available as a deployable application. - Control and data planes: Distinguish the Workflow Service control plane from its data plane. Describe the control plane as the platform service that deploys and operates workflows, and the data plane as SDK code inside each workflow container that handles each HTTP request and invokes the developer's function.
Why it matters: Separating operations around requests from execution during requests clarifies where workflow deployment logic and application request logic belong. - Workflow lifecycle responsibilities: Describe the two groups of Workflow Service responsibilities: deployment and operations around request time, and request handling at request time. Include representative deployment tasks—packaging code into a container, creating replicas, registering API Gateway routes, monitoring health, and scaling—and representative request tasks—receiving HTTP, invoking developer code, coordinating platform-service calls, and returning a response.
Why it matters: Naming the concrete responsibilities makes the control-plane/data-plane boundary actionable when implementing the service.
Key terms: Workflow Service, workflow, platform services, gRPC contract, SDK client, control plane, data plane, workflow container, API Gateway, SDK
9.1 Workflow runtime server
This section introduces the SDK-embedded workflow runtime server that starts in each workflow container and exposes a decorated Python function as an HTTP service.
- Workflow runtime server role: Explain that the workflow runtime server listens for HTTP requests, invokes the developer's workflow function, and returns responses, thereby turning a decorated Python function into a production HTTP service.
Why it matters: The runtime server is the component that makes workflow code callable in production. - Runtime server packaging model: Describe that the runtime server is library code shipped with the SDK rather than a separate service, and that the deployment pipeline configures it as the workflow container's entry point.
Why it matters: Embedding the runtime with each workflow lets a started container immediately run the server that exposes its own code.
Key terms: workflow runtime server, SDK, @workflow decorator, HTTP service, workflow function, runtime server, GenAIPlatform class, service clients, container entry point
9.1.1 The @workflow decorator
This section explains how the @workflow decorator preserves a plain Python workflow function while attaching deployment and runtime metadata that the platform consumes at different lifecycle stages.
- Decorator metadata model: Explain how @workflow attaches a _workflow_metadata dictionary to a workflow function without changing the function's normal calling behavior. Describe the deployment pipeline and runtime server as separate consumers of that same metadata at deployment time and container startup.
Why it matters: A single developer declaration can configure infrastructure and runtime behavior while leaving workflow code simple and locally runnable. - Scaling and resource configuration: Describe the scaling and resource metadata declared by @workflow, including replica bounds, CPU or memory utilization thresholds, per-container CPU and memory allocation, and optional GPU requests. Explain when memory-based scaling is a better saturation signal than CPU-based scaling.
Why it matters: Correct scaling signals and resource allocations keep workflows responsive without provisioning unsuitable or wasteful infrastructure. - Reliability metadata enforcement: Explain how timeout_seconds limits one request and returns a 504 when the runtime server terminates an overlong execution, and how max_retries configures automatic exponential-backoff retries for transient gRPC platform-service failures. Distinguish the retry choices appropriate for a tight synchronous workflow from those suitable for a long-running asynchronous workflow.
Why it matters: Timeouts and retry policies prevent stuck requests while allowing workflows to recover from brief network or gateway failures. - Direct invocation and async rejection: State why a decorated workflow can still be invoked directly for local development and testing, and why @workflow rejects an async def function at import time.
Why it matters: Keeping workflows directly callable simplifies testing, while early async-function rejection avoids a runtime failure caused by worker-thread execution.
Key terms: @workflow, _workflow_metadata, deployment pipeline, runtime server, min_replicas, max_replicas, target_cpu_percent, target_memory_percent, cpu, memory
9.1.2 From decorated function to HTTP server
This section explains how the SDK runtime turns a decorated Python workflow into an independently deployed HTTP service and selects HTTP handling based on its response mode.
- Runtime server transformation: Explain how the workflow runtime server turns a decorated Python function into an HTTP endpoint when its container starts. Describe that the runtime is SDK library code used as the container entry point, rather than a separate service.
Why it matters: This explains how developer-written workflow code becomes a callable production service without the developer building an HTTP server. - Workflow container selection: Describe how the runtime server uses the WORKFLOW_NAME environment variable to identify the workflow that a container serves. Explain why multiple decorated functions in one Python file still become separate container images and independently scalable, deployable, and versioned services.
Why it matters: Selecting exactly one workflow per container preserves independent lifecycle control even when developers organize several workflows in one source file. - Response mode translation: Distinguish how the runtime server translates synchronous, streaming, and asynchronous workflow outputs into HTTP responses. State that metadata response_mode selects a different handler because each interaction pattern has different HTTP response mechanics.
Why it matters: Choosing the correct handler lets one workflow interface support immediate results, progressive output, and long-running work correctly.
Key terms: runtime server, SDK, container entry point, @workflow, api_path, FastAPI, find_workflow, WORKFLOW_NAME, one-workflow-one-service, container image
9.1.3 Synchronous mode
Synchronous mode runs short workflows in a single open HTTP request while preserving tracing, validating inputs, enforcing end-to-end timeouts, and returning safe structured errors.
- Synchronous mode suitability: Describe when synchronous mode is appropriate and when streaming or asynchronous mode should replace it. State that the HTTP connection remains open while the workflow runs.
Why it matters: Choosing the response mode determines whether users get a responsive experience or wait too long on an open request. - Automatic trace-context propagation: Explain how the synchronous handler extracts distributed trace context from request headers, places it in a context variable, and lets a GenAIPlatform instance automatically carry that context into session, data, and model calls.
Why it matters: Automatic propagation links the workflow and all of its downstream platform calls into one distributed trace without burdening workflow authors. - Request validation and safe errors: Describe how the handler validates JSON request arguments against the workflow function signature and how it maps invalid input and unhandled exceptions to safe HTTP JSON errors.
Why it matters: Early validation gives callers useful input errors, while generic exception responses avoid exposing sensitive implementation details. - End-to-end timeout handling: Explain how the runtime applies the decorator's timeout_seconds to the whole workflow with asyncio.wait_for, runs synchronous code in a separate thread, and propagates the remaining time budget as downstream gRPC deadlines.
Why it matters: A single end-to-end deadline prevents slow workflows from monopolizing HTTP connections and prevents downstream work from finishing after the caller has already timed out.
Key terms: synchronous mode, streaming mode, asynchronous mode, HTTP connection, distributed trace context, request headers, context variable, GenAIPlatform, platform.sessions, platform.data
9.1.4 Streaming mode
This section explains how streaming workflows progressively deliver model tokens as SSE events while safely detecting completion, timeouts, and errors.
- Streaming workflow pattern: Describe how a workflow is written and configured for streaming: it sets response_mode="stream", calls platform.models.chat_stream(), iterates over progressive tokens, and yields each value rather than returning one complete response.
Why it matters: This pattern lets an application send model output to the user as it is generated. - SSE stream handling: Explain how the runtime server turns a streaming workflow generator into Server-Sent Events: it validates and traces the request, serializes each yielded dictionary as JSON, and emits it with the SSE data: prefix and double-newline event boundary.
Why it matters: The handler is the bridge between developer-produced chunks and an HTTP client that can receive them immediately. - Chunk timeout and completion: Explain why the streaming handler obtains generator chunks with next(gen, _DONE) inside asyncio.wait_for: the timeout detects a stalled model service per chunk, while the _DONE marker signals normal generator exhaustion without a StopIteration error in a task. Describe that mid-stream failures are sent as structured error events before the stream closes.
Why it matters: A production stream must distinguish successful completion from a stalled or failed generation instead of leaving clients waiting or reporting false errors. - Perceived latency benefit: Justify streaming as a responsiveness improvement: full generation can take the same total time as synchronous mode, but showing text within a second or two lowers perceived latency and makes the interaction feel conversational.
Why it matters: Streaming improves the user's experience without requiring the model to finish generating sooner.
Key terms: response_mode="stream", yield, platform.models.chat_stream(), iterator, tokens, Server-Sent Event, SSE, event_stream, data:, JSON
9.1.5 Asynchronous mode
This section explains how asynchronous workflow mode immediately returns a pollable job while the runtime executes long-running work in the background and the Workflow Service persists its state.
- Asynchronous execution flow: Trace what happens when a workflow uses response_mode="async", from the incoming HTTP request through job creation, background execution, and the immediate HTTP 202 response.
Why it matters: This pattern lets clients start workflows lasting minutes or hours without holding an HTTP connection open. - Job status and failure reporting: Describe how the asynchronous handler and background runner record processing, progress, completion, and failure so polling clients can observe a job's outcome.
Why it matters: Persisted status and meaningful failures prevent clients from waiting indefinitely for long-running work. - Runtime and service responsibilities: Distinguish the workflow container's responsibility for executing developer code from the Workflow Service's responsibility for owning persistent job state and serving the polling URL.
Why it matters: Separating execution from job tracking keeps long-running workflow handling clean and durable across failures.
Key terms: response_mode="async", job record, job ID, background task, asyncio.create_task(), HTTP 202 Accepted, create_job(), processing, update_job_progress(), complete_job()
9.2 Async job lifecycle
This section introduces how asynchronous workflow jobs are stored, exposed through polling, and updated with developer-reported progress.
- Async job lifecycle questions: State the two unresolved aspects of asynchronous execution: where the job record created by platform.workflows.create_job() lives and contains, and how the /jobs/{job_id} polling endpoint serves status before completion.
Why it matters: These questions define the lifecycle information a client needs after it submits an asynchronous workflow. - Polling progress reporting: Describe the planned mechanism by which workflow developers report progress while a workflow executes, so polling clients can receive progress information between submission and completion.
Why it matters: Progress reporting lets clients see that a long-running asynchronous workflow is advancing rather than merely waiting for a final result.
Key terms: async handler, platform.workflows.create_job(), job record, /jobs/{job_id}, progress reporting, polling clients, workflow executes
9.2.1 Job storage and the polling endpoint
This section defines the persistent job record and external polling endpoint that let asynchronous workflows report progress, return outcomes, and recover orphaned work.
- Job record consumer requirements: Explain how the polling client, runtime server, and crash recovery logic each use the same job record, and identify the information each consumer needs.
Why it matters: A job record must support both normal asynchronous execution and recovery when a workflow container disappears. - Universal jobs table design: Describe the jobs table fields and access patterns that support asynchronous workflow execution, including lifecycle status, progress, generic JSON payloads, and recovery queries.
Why it matters: A universal persistent record lets workflows with different request and response shapes use one reliable asynchronous execution mechanism. - Polling endpoint lifecycle: Trace the asynchronous job lifecycle from job creation through the 202 response and repeated GET requests to /jobs/{job_id}, including how polling responses differ for processing, completed, and failed jobs.
Why it matters: Polling gives external clients a simple way to observe long-running workflow work without holding the original request open.
Key terms: polling client, runtime server, crash recovery logic, job record, orphaned job, assigned endpoint, job_id, status, processing, completed
9.2.2 Progress reporting
This section explains how asynchronous workflows report progress, persist checkpoints for crash recovery, and record completion or failure through the Workflow Service.
- Context-based progress reporting: Explain how an asynchronous workflow updates a job's progress without the developer supplying a job ID. Describe how the SDK obtains the job ID from a context variable and how a later poll returns the updated progress message.
Why it matters: This lets workflow authors expose meaningful status updates while the platform preserves the connection to the correct asynchronous job. - Checkpoint-based crash recovery: Describe how save_checkpoint and load_checkpoint let an asynchronous workflow resume after a crash. Explain that saved intermediate results are JSON-serialized into the job record, then returned when an orphaned job is re-dispatched to a new container with the same input.
Why it matters: Checkpointing avoids repeating completed expensive work, such as LLM calls, after a long-running workflow fails. - Async job terminal state updates: Describe how the runtime records successful and failed asynchronous workflow executions. State that complete_job stores the serialized return value and marks the job completed, while fail_job marks it failed and stores the error message.
Why it matters: A polling client needs a persisted terminal status and either a result or error to determine what happened to its asynchronous request. - Simple job-record writes: Explain why job persistence is deliberately limited to single-column updates by primary key. Relate that simplicity to the fact that asynchronous workflow coordination belongs in the runtime server rather than the database layer.
Why it matters: Keeping storage simple concentrates coordination logic in the runtime component designed to manage asynchronous execution.
Key terms: update_job_progress, _job_id_var, context variable, Workflow Service, /jobs/{job_id}, save_checkpoint, load_checkpoint, checkpoint_json, orphaned job, crash re-dispatch
9.3 Production readiness: concurrency, reliability, and health
This section introduces the production concerns the workflow runtime server must handle for AI applications: concurrent requests, retries for transient failures, and container health checks.
- Production runtime concerns: Describe the three production concerns handled by the workflow runtime beyond basic request processing: concurrency for simultaneous requests, automatic retry for transient network failures, and health endpoints for container lifecycle management.
Why it matters: These capabilities let a workflow service keep operating when real production traffic and failures replace isolated requests. - AI-specific production pressure: Explain why AI workflows make standard concurrency and reliability patterns especially important: LLM calls can hold connections for seconds or minutes, and third-party providers can impose rate limits, capacity constraints, and transient timeouts.
Why it matters: AI workloads can turn ordinary web-service weaknesses into connection exhaustion, event-loop starvation, and provider-related failures. - Production versus isolated requests: Distinguish serving a single workflow request in isolation from operating the runtime in production, where many requests can arrive together, network calls can fail, and the platform must assess container health.
Why it matters: This distinction explains why a working HTTP endpoint alone is not a production-ready workflow service.
Key terms: runtime server, concurrency, automatic retry, transient network failures, health endpoints, container lifecycle, LLM providers, connection holding, event-loop starvation, rate limits
9.3.1 Concurrency inside a single container
This section explains how a workflow runtime handles many simultaneous requests within containers and scales safely across replicas under load.
- Event-loop and worker concurrency: Explain how Uvicorn's asyncio event loop multiplexes requests that are waiting on platform-service gRPC calls, and how multiple worker processes increase parallel request handling inside one container.
Why it matters: This explains why I/O-heavy AI workflows can serve many users without requiring one CPU core per request. - Execution modes under load: Describe how synchronous, streaming, and asynchronous workflows are dispatched so that the runtime can continue handling other requests.
Why it matters: The execution mode determines whether the runtime waits for a result, streams chunks, or returns a job ID immediately. - Request state isolation: Explain why simultaneous requests in the same container do not contaminate each other's platform clients, retry state, trace context, or job IDs.
Why it matters: Isolation ensures that concurrent users retain correct tracing and request-specific behavior even when their workflows share a container. - Replica scaling and overload protection: Trace how the API Gateway distributes traffic across healthy replicas, how CPU-threshold scaling adds replicas, and how rate limiting protects existing containers during a scaling lag.
Why it matters: Scaling takes time, so load balancing and rate limiting prevent a traffic burst from overwhelming the capacity already running. - Container capacity estimates: State the section's approximate concurrency estimates and why I/O-waiting workflows can achieve them despite limited CPU.
Why it matters: Capacity estimates provide a starting point for reasoning about how worker count and replica count affect throughput.
Key terms: uvicorn, ASGI server, asyncio event loop, worker process, thread pool, gRPC, asyncio.to_thread, sync handler, streaming workflow, generator
9.3.2 Retry at the network layer
This section explains how SDK-level gRPC retries make workflow-to-platform-service calls resilient to transient network failures without changing workflow code.
- SDK retry scope: Distinguish SDK network-layer retries from the Model Service retries used for external model providers. Explain that SDK retries cover calls from a workflow container through the gateway to platform services, while Model Service retries cover calls from the Model Service to providers such as OpenAI or Anthropic.
Why it matters: Separating these layers makes it possible to diagnose where a failed model-related request actually failed. - Interceptor retry behavior: Describe how a gRPC retry interceptor wraps outgoing SDK calls, returns successful responses normally, and retries only designated transient failures. State that UNAVAILABLE, DEADLINE_EXCEEDED, and RESOURCE_EXHAUSTED are retryable because they represent temporary unreachability, timeout, or overload, whereas invalid requests propagate immediately.
Why it matters: Retrying the right failures masks brief infrastructure disruptions without repeatedly sending requests that cannot succeed. - Backoff and inherited configuration: Explain that retry attempts use exponential backoff of 100, 200, then 400 milliseconds to avoid hammering a struggling service, and that BaseClient attaches the interceptor to its gRPC channel with grpc.intercept_channel. Describe how a workflow's decorator metadata supplies max_retries once, causing all SDK service clients in that workflow to inherit that retry limit.
Why it matters: Centralized backoff and configuration provide consistent resilience across service calls while keeping synchronous workflows responsive.
Key terms: SDK retry, network layer, Model Service retry, workflow container, gateway, platform services, gRPC interceptor, outgoing call, UNAVAILABLE, DEADLINE_EXCEEDED
9.3.3 Health endpoints
This section explains how separate shallow liveness and readiness endpoints let the orchestrator manage workflow containers without turning transient dependency failures into outages.
- Liveness and readiness roles: Distinguish the /health/live liveness probe from the /health/ready readiness probe, including what a successful HTTP response means for each and how the orchestrator acts on failure.
Why it matters: Using the two probes for different decisions lets the platform recover broken processes without unnecessarily removing usable containers from traffic. - Startup and shutdown traffic control: Describe how readiness controls traffic during startup and graceful shutdown: it stays unavailable while initialization occurs, then returns 503 on termination so in-flight requests can finish before the container exits.
Why it matters: Readiness-based traffic draining prevents requests from being sent to containers that are initializing or about to stop. - Shallow health-check design: Justify keeping liveness and readiness probes shallow by checking only that the server process is responsive, while handling temporary dependency connectivity failures with per-request SDK retry logic.
Why it matters: Shallow probes prevent a brief shared-dependency outage from causing every workflow container to be removed from service at once.
Key terms: /health/live, liveness probe, /health/ready, readiness probe, container orchestrator, in-flight requests, graceful shutdown, termination signal, load balancing pool, HTTP 503
9.4 Workflow composition
This section introduces workflow composition as calling independently deployed workflows over their HTTP APIs instead of duplicating their logic.
- Workflow composition model: Explain how a parent workflow can compose capabilities by calling other deployed workflows through the API Gateway, using the same HTTP APIs available to external clients.
Why it matters: Composition lets a workflow reuse platform capabilities without embedding their implementation. - Composition over duplication benefits: Justify choosing workflow composition over copying search and summarization logic by describing independent updates, isolated dependencies and resource requirements, scaling, and team ownership.
Why it matters: Keeping capabilities independent allows teams to improve and operate them without forcing every consumer to copy their code or infrastructure needs.
Key terms: workflow composition, parent workflow, child workflows, API Gateway, HTTP APIs, duplication, composition, dependencies, scaling, team ownership
9.4.1 Calling other workflows
This section shows how a parent workflow invokes an independently deployed child workflow through the SDK and API gateway.
- Child workflow invocation: Describe how a parent workflow uses platform.workflows.call() with a child workflow api_path and request body to invoke that workflow and receive its result as a dictionary.
Why it matters: This lets a workflow reuse another workflow's capability without embedding that capability in its own code. - Gateway-routed workflow execution: Trace the path of a child-workflow call: the SDK sends an HTTP POST through the gateway to the child's api_path, the gateway routes it to a container running that workflow, and the parent continues after the JSON response returns.
Why it matters: Understanding this request path explains why composed workflows remain separate deployable services rather than becoming one shared process. - Independent workflow deployment: Explain why a child workflow remains independently operated when called by a parent: it has its own containers, dependencies, scaling rules, retry configuration, and Workflow Service registry entry, so a new child version can be deployed without redeploying the parent.
Why it matters: Independent deployment lets teams evolve reusable workflow capabilities without disrupting the workflows that consume them.
Key terms: platform.workflows.call(), api_path, request body, parent workflow, child workflow, HTTP POST, gateway, /knowledge/search, JSON response, container
9.4.2 Parallel workflow calls
This section explains how a parent workflow can call independent child workflows in parallel, the behavior and limits of that composition model, and when more formal orchestration is warranted.
- Parallel child workflow calls: Explain when to use workflow.call_parallel() instead of sequential workflow.call() calls and describe how it runs independent child workflows concurrently. State that the SDK waits for all calls and returns their results in the same order as the input list.
Why it matters: Parallelizing independent work avoids making a parent workflow wait unnecessarily for one child before starting another. - Parallel failure and response handling: Describe how call_parallel() handles child failures and response modes: each call uses the normal call() path with its retries and response handling, including polling async children; a child failure after retries propagates an exception to the parent immediately.
Why it matters: A parent can rely on the normal call behavior for retries and async children while retaining responsibility for handling a terminal child failure. - Dynamic parallel fan-out: Describe how a workflow can build the plain Python list passed to call_parallel() dynamically, such as creating one child call per discovered document. Explain that the number of parallel calls can vary with the input and child containers scale to meet that workload.
Why it matters: Dynamic fan-out lets a workflow process however many independent items it discovers without hardcoding a fixed number of child calls. - Code-defined composition tradeoffs: Distinguish code-sequenced composition with call() and call_parallel() from a dedicated workflow orchestrator. Explain that ordinary Python makes dependency flow implicit and readable, but the platform cannot see the graph to retry individual steps, checkpoint between calls, or show step-level progress; complex shared-dependency or expensive-step pipelines may need Argo or Metaflow.
Why it matters: Knowing the visibility limits of code-defined composition helps teams choose a dedicated orchestrator before a complex pipeline becomes difficult to recover or operate. - Agent loops within workflows: Explain why an agentic workflow can remain inside one workflow function as a while loop: the model chooses an action, the platform executes it, and the loop repeats until a final answer. State that this iterative reasoning logic is invisible to the platform, which sees only the workflow's HTTP request and response.
Why it matters: This separates agent-control logic owned by application code from the request-and-response behavior managed by the platform.
Key terms: workflow.call_parallel(), workflow.call(), child workflow, parent workflow, gateway, retry logic, response mode handling, async, polls, exception propagates
9.4.3 Response mode handling
This section explains how SDK workflow calls hide child response modes, handle HTTP reliability, and when workflow composition is worth its network cost.
- Transparent child response handling: Explain how platform.workflows.call() inspects a child workflow's HTTP response mode and hides sync, async, and streaming differences from the parent developer. Describe the return path for a 200 JSON sync response, a 202 async response, and a text/event-stream response.
Why it matters: A parent workflow can keep the same calling code even when a child workflow changes how it delivers results. - HTTP retry classification: Describe how workflow-to-workflow HTTP calls are retried at the network layer. Distinguish transient connection failures and 5xx responses, which are retried, from 4xx client errors, which are not retried.
Why it matters: Retrying only failures that may resolve prevents brief network or scaling disruptions from breaking composition without repeatedly sending invalid requests. - Async polling and stream completion: Trace how the SDK finishes asynchronous and streaming child calls: polling continues until completed, failed, or cancelled, while Server-Sent Event data chunks are parsed and collected and the final done signal is excluded.
Why it matters: These helpers turn long-running jobs and incremental output into usable child-workflow results without exposing protocol work to parent authors. - Composition boundary decisions: Justify choosing workflow composition for real boundaries such as shared cross-team logic, dependency isolation, or independent scaling, rather than composing simple logic with shared dependencies under one team's ownership. State the network-hop and gateway-round-trip cost of each child call.
Why it matters: Using composition only where boundaries are real avoids needless latency and architectural complexity.
Key terms: platform.workflows.call(), sync response mode, async response mode, streaming response mode, text/event-stream, job ID, _http_post, HTTP retries, 5xx errors, 4xx errors
9.5 The Workflow Service contract
This section introduces the Workflow Service’s multi-consumer gRPC contract and organizes its operations into registry, deployment, and job groups.
- Multi-consumer contract design: Explain why the Workflow Service contract differs from other platform service contracts by serving three consumers rather than only workflow code using an SDK client. Identify the deploy CLI, the runtime server in workflow containers, and external clients as those consumers.
Why it matters: The contract must accommodate distinct callers, access patterns, and trust levels instead of being designed only for an in-workflow SDK client. - Three operation groups: Describe how the Workflow Service contract maps its three consumers' needs into registry, deployment, and job operations. State the major responsibilities covered by each group.
Why it matters: Grouping operations by lifecycle responsibility makes it clear which caller manages workflow definitions, deployments, and asynchronous jobs.
Key terms: Workflow Service, gRPC contract, genai-platform deploy CLI, runtime server, external clients, SDK client, registry operations, deployment operations, job operations, workflow specification
9.5.1 Registry operations
This section defines how the Workflow Service registry records workflow specifications, supports their lifecycle operations, and supplies deployment infrastructure settings.
- Workflow registration specification: Describe how a workflow enters the platform through RegisterWorkflow and identify the key contents of its WorkflowSpec.
Why it matters: The registry specification is the information the platform needs to deploy and manage a workflow consistently. - Resource and GPU configuration: Explain how ResourceConfig and GPU fields translate a workflow's resource requirements into container limits and Kubernetes node scheduling.
Why it matters: Resource configuration ensures a workflow receives the CPU, memory, and accelerator capacity it requires when deployed. - Registry lifecycle operations: Describe the retrieval, listing, update, and deletion operations for registered workflows, including the effects of updating and deleting a workflow.
Why it matters: These operations let platform tooling and administrators inspect and change workflows throughout their lifecycle. - Deployment version tracking: State how the workflow version field is managed and why the platform keeps it.
Why it matters: Deployment versions provide the basis for rolling a workflow back to an earlier deployment.
Key terms: registry, RegisterWorkflow, WorkflowSpec, api_path, response_mode, scaling configuration, ResourceConfig, cpu, memory, GPU
9.5.2 Deployment operations
This section defines how the Workflow Service deploys registered workflows, tracks their running state, routes traffic, rolls back versions, and formalizes async job operations.
- Deployment realization process: Explain how DeployWorkflow turns a registered WorkflowSpec into running containers and API Gateway routes, unlike RegisterWorkflow, which records the intended workflow without starting it.
Why it matters: A registered workflow is only a specification until deployment creates the runnable service behind it. - Deployment state and routing: Describe how a WorkflowDeployment represents desired and actual replica state, and how healthy_endpoints drives API Gateway routing as replicas are created or removed.
Why it matters: Traffic should reach only healthy running containers, while operators need to see whether actual capacity matches the requested scale. - Deployment status and rollback: Trace deployment status from deploying to healthy or failed, and explain that RollbackWorkflow redeploys an older registered WorkflowSpec using that version's container image.
Why it matters: Status makes deployment outcomes visible to operators, and rollback provides recovery when a new version cannot be safely used. - Async job management contract: State that the gRPC job-management operations formalize the existing async job implementation: WorkflowJob mirrors a jobs-table row, and each request maps to one column update.
Why it matters: The contract exposes async job lifecycle changes consistently without inventing a separate data model from the implementation.
Key terms: RegisterWorkflow, DeployWorkflow, WorkflowSpec, API Gateway, WorkflowDeployment, desired_replicas, current_replicas, healthy_endpoints, routing table, GetDeploymentStatus
9.6 Deployment pipeline
This section introduces the deployment pipeline details: how the CLI builds and publishes a workflow container and makes its API route available through the gateway.
- Deployment pipeline overview: Describe the end-to-end deployment flow introduced here: a developer runs genai-platform deploy, which builds a container image, pushes it to a registry, registers the workflow with the Workflow Service, and results in the workflow API path being routed by the gateway.
Why it matters: This is the path that turns a workflow implementation into an API that applications can reach.
Key terms: genai-platform deploy, container image, registry, Workflow Service, API gateway, routing table
9.6.1 What genai-platform deploy does
This section explains how genai-platform deploy packages a decorated workflow, registers and deploys it, and promotes one tested image across environments.
- Six-step deployment pipeline: Trace the six steps performed by genai-platform deploy: discover decorated workflows, read their metadata, generate or use a Dockerfile, build and push an image, register a WorkflowSpec, and trigger deployment while polling for its result.
Why it matters: This clarifies which deployment responsibilities belong to the CLI and which belong to the Workflow Service. - Container runtime contract: Describe how the generated Dockerfile uses cached layers and environment variables to start the SDK runtime server for a particular workflow. State the minimum custom-Dockerfile requirements: install genai-platform-sdk, set WORKFLOW_NAME, and use the runtime server as the entrypoint.
Why it matters: The container contract is what lets the platform locate the workflow function and serve it consistently after deployment. - Environment-aware deployment: Explain what the --env flag changes: the target registry and cluster, the gateway URL baked into the image, and the environment tag on workflow registration that keeps routing isolated between environments.
Why it matters: Environment isolation lets teams test a workflow before it receives production traffic. - Build-once promotion discipline: Justify build-once, promote-many deployment: build an image once, then use promote to redeploy that identical version to later environments without rebuilding, while allowing environment-specific targets, gateway endpoints, and scaling settings.
Why it matters: Promoting the same artifact ensures that the exact binary tested in staging is the one that reaches production.
Key terms: genai-platform deploy, @workflow, _workflow_metadata, WorkflowSpec, RegisterWorkflow, DeployWorkflow, Dockerfile, WORKFLOW_MODULE, WORKFLOW_NAME, GENAI_GATEWAY_URL
9.6.2 Route registration with the API gateway
This section explains how the Workflow Service keeps the API Gateway’s routes mapped to the healthy container endpoints of each deployed workflow throughout its lifecycle.
- Gateway route registration: Explain how the Workflow Service registers a deployed workflow with the API Gateway after its container replicas pass readiness probes. State that it sends the workflow’s api_path and its healthy endpoint addresses so gateway requests can be forwarded to that workflow.
Why it matters: A workflow is reachable through its API path only when the gateway has a route to its healthy running replicas. - Dynamic route lifecycle: Describe how the Workflow Service updates the API Gateway routing table as replicas are added, removed, replaced, or deleted. Explain that deleting a workflow removes its route entirely, so requests to its former API path receive a 404 response.
Why it matters: Keeping routes synchronized with healthy replicas prevents the gateway from sending traffic to failed, drained, or nonexistent workflow containers.
Key terms: Workflow Service, API Gateway, routing table, api_path, WorkflowDeployment, readiness probes, auto-scaling, readiness probe, health check, drained replicas
9.7 Runtime management
This section explains how the Workflow Service translates workflow decorator settings into Kubernetes runtime resources that maintain availability, autoscale capacity, and safely roll out updates.
- Kubernetes runtime resources: Describe the two Kubernetes resources the Workflow Service generates for each workflow and what the Kubernetes Deployment is responsible for at runtime.
Why it matters: This explains how a deployed workflow stays available without developers managing Kubernetes manifests directly. - CPU-based autoscaling behavior: Explain how the HorizontalPodAutoscaler changes workflow replica counts from CPU utilization, including the min/max bounds and asymmetric scale-up and scale-down stabilization windows.
Why it matters: Autoscaling lets workflows absorb traffic spikes while avoiding unnecessary capacity and unstable replica changes during fluctuating load. - Metric-driven rollout rollback: Trace how an updated workflow is rolled out and rolled back: Kubernetes performs a rolling pod replacement, while the Workflow Service compares application error rates with the prior version baseline to decide whether to revert.
Why it matters: The platform protects users from unhealthy releases using application-level evidence, not merely the fact that containers are running.
Key terms: Workflow Service, Kubernetes Deployment, HorizontalPodAutoscaler, @workflow decorator, min_replicas, resource parameters, CPU utilization, target_cpu_percent, max_replicas, stabilization windows
9.8 Summary
This summary connects the Workflow Service’s runtime, asynchronous execution, composition, and deployment control plane into one production architecture for AI workflows.
Where this is used later
- Deploying Claw (chapter 10) · Deploying the completed assistant relies on the Workflow Service to expose its workflow as an application.
- The simplest assistant: a model call in a workflow (chapter 10) · A decorated workflow function becoming an HTTP-serving runtime.
- Seeing what Claw does: observability (chapter 10) · Observability of the complete assistant depends on correlated tracing across its platform calls.
- Streaming the response (chapter 10) · The assistant streams its generated response to the client.
- Grounding Claw in reality: knowledge retrieval and agentic RAG (chapter 10) · The assistant can combine knowledge-retrieval capabilities within its workflow.
- What the platform gave us (chapter 10) · The final platform benefits include composing managed capabilities rather than rebuilding them.
- The complete agent loop (chapter 10) · The complete agent loop builds on understanding which work can be composed as workflow calls versus performed within workflow code.
- From conversation to action: tools and the agent loop (chapter 10) · The assistant's transition from conversation to action uses tools and an agent loop.
Concepts explained in this chapter
- workflow runtime serverA workflow runtime server is SDK library code that imports a workflow module, registers an HTTP route, receives requests, invokes a workflow function, and returns its responses as a service.
- retryA retry is a repeated attempt to perform an operation after a transient failure or unsuccessful attempt.
- job idA job ID is a unique identifier assigned to an asynchronous workflow execution so clients and services can refer to its job record.
- readiness checkA readiness probe is an automated check that determines whether a service instance is initialized and ready to receive traffic.
- response modeA response mode is a workflow configuration that determines whether results are returned synchronously or incrementally as a stream.
- context variableA context variable is a dynamically scoped variable whose value can differ across execution contexts such as threads or asynchronous tasks.
- job recordA job record is the Workflow Service's stored representation of one submitted workflow execution, including its status, payloads, progress, result, error, and assigned container.
- pollingPolling is the repeated checking of an endpoint or resource until a desired state or result is available.
- deployment pipelineA deployment pipeline is an automated sequence of steps that builds, packages, and releases software for execution in a target environment.
- interceptorA gRPC interceptor is a client- or server-side component that intercepts remote procedure calls to apply behavior before or after invocation.
- rollbackA rollback is the restoration of a deployed system to a previously known version or configuration after a problematic change.
- workflow serviceA workflow service is a platform service that coordinates sequential and parallel operations, state management, retries, and failure recovery in complex processes.
- workflow compositionWorkflow composition is the pattern of building an application workflow by invoking other independently deployed workflows instead of duplicating their logic.
- call_parallelcall_parallel is a workflow API operation that invokes multiple child workflows concurrently and returns their results in submission order.
- deploy CLIThe deploy CLI is a command-line tool that packages workflow code, publishes its container image, and registers the workflow for deployment.
- workflow decoratorA workflow decorator is a function annotation that converts an ordinary function into a managed workflow process.
- concurrencyConcurrency is the management of multiple in-progress operations so that their execution can overlap without requiring simultaneous CPU execution.
- control planeA control plane is the part of a distributed system responsible for deployment, configuration, operations, and management rather than handling application requests.
- data planeA data plane is the part of a distributed system that processes application requests and carries out runtime operations.
- UvicornUvicorn is an ASGI server that listens for network connections and dispatches incoming requests to an ASGI application.
- liveness checkA liveness probe is a health-check interface that indicates whether a running service instance is alive and able to be managed by infrastructure.
- workflowA workflow is a defined sequence of dependent operations that coordinates processing steps and manages their execution state.
- child workflowA child workflow is a workflow invoked by another workflow and executed as an independently managed deployment.
- parent workflowA parent workflow is a workflow that invokes and coordinates one or more other workflows.
- recoveryCrash recovery is the detection and re-dispatch of work whose executing process or container has unexpectedly disappeared.
- asynchronous programmingAsynchronous execution is an execution model in which a request returns before its operation completes, allowing processing to continue independently in the background.
- health checkA health check is an automated test that determines whether a running service instance is functioning and able to receive work.
- health check endpointA health endpoint is an API endpoint that reports whether a service process is running or ready to receive traffic.
- server-sent eventsServer-Sent Events is an HTTP-based mechanism for continuously delivering unidirectional, incrementally generated events from a server to a client.
- stream processingA streaming response is an output delivery mode that sends generated content incrementally as it becomes available instead of waiting for the complete result.
- exponential backoffExponential backoff is a retry timing strategy that increases the delay between successive attempts, typically by a multiplicative factor.
- synchronous callSynchronous execution is an execution model in which the caller waits for an operation to complete before receiving its result.
- gRPC contractA gRPC contract is a formal service interface that defines the operations and interaction structure exposed between gRPC services.
- evaluationEvaluation is the systematic assessment of an AI system or change against defined evidence and criteria.
- experimentation-as-a-service platformAn experimentation service is a platform service that provides infrastructure for running controlled experiments and measuring the effects of application changes.
- human-in-the-loopHuman review is the inspection and validation of an item by a person before it is accepted for an intended purpose.
- offline evaluationOffline evaluation assesses an AI system or change using previously collected datasets rather than live interactions.
- online evaluationOnline evaluation assesses an AI system or change using live production interactions and their outcomes.
- asyncio.to_threadasyncio.to_thread is a Python API that runs a synchronous callable in a separate thread while allowing the calling coroutine to remain asynchronous.
- isolationIsolation is the separation of software workloads so that their execution, resources, and interactions are constrained from affecting one another.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
