liveBook · Manning

API gateway

As explained in Designing AI Systems

An API gateway is a gateway component that provides a unified entry point for accessing backend services and applies shared cross-cutting controls.

An API gateway provides a stable entry point to backend services while centralizing controls that would otherwise be duplicated across workflows. It authenticates requests, enforces rate limits, terminates TLS, and routes traffic to healthy containers. Services can deploy and scale independently without changing the client-facing endpoint. section 2.4.1 section 2.6

For internal calls, the gateway also acts as a gRPC proxy. SDK clients connect to it and identify the target service in request metadata; the gateway routes the call and forwards Protocol Buffer data without deserializing it. Its service registry tracks available replicas, so workflows do not need configuration changes when services scale. section 2.4.3 section 2.5.4

Figure 2.5 The Request Flow: A user's question flows from the mobile app through the API Gateway (over HTTPS), reaches the workflow container, which then makes multiple internal gRPC calls through the gateway to reach platform services. The gateway acts as a gRPC proxy, routing internal service communication while maintaining Protocol Buffer efficiency.

Follow an HTTPS request into a workflow and its internal gRPC calls through the gateway. section 2.4.3

The gateway supports production operations across independently deployed services:

  • Distribute requests across healthy replicas and stop routing to failed containers. section 2.4.1
  • Shift traffic gradually during deployments, with rollback if problems appear; let old containers finish in-flight requests before shutdown. section 2.4.1 section 2.4.4
  • Scale gateway capacity horizontally with stateless instances behind a load balancer; each instance routes using its own registry view. section 2.4.4
  • Convert model-service gRPC streams to Server-Sent Events for external clients, forwarding chunks as they arrive. section 3.3.1

Connecting a workflow SDK

A platform client resolves its gateway address in three steps: use an explicitly supplied URL, otherwise read GENAI_GATEWAY_URL, otherwise fall back to localhost:50051. This lets the same workflow code connect in development and production. section 2.5.1

Listing 2.16 GenAIPlatform constructor with gateway configuration
class GenAIPlatform:
    def __init__(self, gateway_url=None):    #A
        if gateway_url:    #B
            self.gateway_url = gateway_url    #B
        else:    #C
            self.gateway_url = os.getenv("GENAI_GATEWAY_URL")    #C
            if not self.gateway_url:    #D
                self.gateway_url = "localhost:50051"    #D

Inspect the Python constructor that implements gateway address precedence. section 2.5.1

How Designing AI Systems covers it

How deep it goes
Traces request routing, scaling, deployments, and SDK address configuration without implementing the gateway itself.
Examples
Traces a patient-assistant request through session, data, and model services, and configures a Python SDK gateway address.
What you can do after
Evaluate an API gateway's role in routing, shared controls, service scaling, and deployment reliability.
How it connects
Connects service discovery, load balancing, authentication, gRPC, Protocol Buffers, and Server-Sent Events.

Also explained in

Compare how 10 books explain API gateway →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.