liveBook · Manning

Service discovery

As explained in Designing AI Systems

Service discovery is the mechanism by which distributed components locate available services and their connection endpoints.

Service discovery lets distributed components locate available services through a shared routing layer. A gateway maintains an internal registry, so workflows call stable gateway endpoints while the gateway selects healthy service instances. section 2.4.3 section 2.4.4

A production-oriented service-discovery pattern combines:

  • A gateway that proxies internal gRPC traffic, preserves Protocol Buffer efficiency, and centralizes routing, authentication, rate limiting, tracing, and authorization. section 2.4.3
  • A registry view maintained by each stateless gateway instance, tracking service replicas and distributing requests across the current set of instances. section 2.4.4
  • Health-aware registration and removal, allowing new instances to receive traffic after becoming healthy while old instances finish in-flight requests before shutdown. section 2.4.4
  • Independent scaling, because services add or remove replicas without requiring workflow configuration changes or affecting unrelated services. section 2.4.4

Routing one request

A mobile request reaches the gateway over HTTPS and enters a workflow container. The workflow makes internal gRPC calls through the same gateway; the registry routes each call to a healthy Session, Model, or Data Service replica. The gateway returns the workflow response to the mobile client while recording tracing and metrics across the request path. section 2.4.3

Figure 2.5 The Request Flow: A user's question flows from the mobile app through the API Gateway (over HTTPS), reaches the workflow container, which then makes multiple internal gRPC calls through the gateway to reach platform services. The gateway acts as a gRPC proxy, routing internal service communication while maintaining Protocol Buffer efficiency.

Trace the complete path from the mobile application through the HTTPS gateway, workflow container, internal gRPC proxy, and platform services. section 2.4.3

How Designing AI Systems covers it

How deep it goes
Explains registry-based routing, health checks, scaling, and deployment behavior, but stops short of implementation code.
Examples
Uses a mobile-to-gateway request, workflow calls, Session Service replicas, and zero-downtime container replacement.
What you can do after
Evaluate how gateways route requests, scale services independently, and update traffic safely during deployments.
How it connects
Connects service discovery with API gateways, service registries, gRPC, health checking, horizontal scaling, and deployment automation.

Also explained in

Compare how 10 books explain service discovery →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.