liveBook · Manning

Data plane

As explained in Designing AI Systems

A data plane is the part of a distributed system that processes application requests and carries out runtime operations.

A data plane is the request-time part of a distributed system. It runs as SDK runtime code inside each workflow container, receives HTTP requests, invokes workflow functions, coordinates platform-service calls, and returns responses. Deployment, scaling, health monitoring, and route registration belong to the separate control plane. section 9.0 section 9.8

A production data plane typically handles:

  • Request translation: discovers the workflow through WORKFLOW_NAME, deserializes JSON bodies into keyword arguments, and serializes return values into JSON. section 9.8
  • Response execution: runs synchronous functions with timeouts, streams generator output through Server-Sent Events, or returns asynchronous job IDs while work continues in a background thread. section 9.8
  • Platform coordination: calls sessions, models, data, and other services through the API Gateway, with retries and exponential backoff for transient network failures. section 9.0 section 9.8
  • Request isolation: creates separate platform instances and copied context variables for concurrent requests, preventing trace and state cross-contamination. section 9.8

Stateless research workflow

An asynchronous research workflow fans out searches across company knowledge, engineering documentation, and the web, merges the results, and synthesizes them with a model call. The data plane returns a job ID immediately while child workflows run in independently scalable containers. Sessions, memory, knowledge bases, and credentials remain in platform services, allowing horizontal scaling. section 10.12

Figure 9.1 Control plane vs. data plane. On the left, the Workflow Service (control plane) connects to Kubernetes to manage container Deployments and to the API Gateway to register routes. On the right, the data plane: multiple workflow containers, each running the SDK's runtime server, receiving HTTP requests from the API Gateway and making gRPC calls to platform services through the gateway. For synchronous and streaming workflows, the control plane and data plane do not communicate during request handling. For asynchronous workflows, the data plane calls the control plane to manage job state.

Use the architecture diagram to trace the boundary: the control plane manages Kubernetes deployments and API Gateway routes, while workflow containers receive requests and call platform services. Asynchronous workflows additionally call the control plane for job state. section 9.0

How Designing AI Systems covers it

How deep it goes
Explains container placement, request modes, tracing, concurrency, retries, and service calls, but defers deployment management to the control plane.
Examples
Demonstrates synchronous, streaming, and asynchronous responses plus a fan-out/fan-in research workflow using parallel child workflows.
What it assumes
Assumes familiarity with workflow functions, containers, HTTP requests, platform services, and the distinction between runtime handling and deployment operations.
What you can do after
Operate request handling with runtime servers, response modes, retries, tracing, concurrency isolation, and horizontally scalable workflow containers.
How it connects
Connects workflow containers with the control plane, API Gateway, platform services, Kubernetes, distributed tracing, and asynchronous job management.

Also explained in

Compare how 5 books explain data plane →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.