liveBook · Manning

Load balancing

As explained in Designing AI Systems

Load-based routing distributes requests among providers or execution paths according to their current workload or capacity.

Load balancing routes each request to the provider with the least current workload instead of concentrating traffic on one provider until failure. This proactively uses aggregate capacity, increases throughput, and reduces rate-limit pressure. section 3.5.3

Implement the routing pattern with a small amount of explicit state:

  • Track active requests independently for every provider. section 3.5.3
  • Select the provider with the lowest current counter when a request arrives. section 3.5.3
  • Increment the selected provider's counter before dispatching the request. section 3.5.3
  • Decrement the counter after completion, regardless of success or failure, so routing state remains accurate. section 3.5.3

Use this strategy when multiple providers expose useful capacity, requests have broadly similar latency, or rate limits make concentrating traffic risky. Prefer equivalent provider accounts or keys when response consistency matters: different model families can produce different outputs, which may be unsuitable for schema-sensitive extraction. section 3.5.3

Operational metrics make the decision measurable. Rising latency, frequent rate-limit fallbacks, provider-specific errors, and throughput changes can indicate that traffic should be redistributed rather than sent repeatedly to one provider. section 3.8.3 section 3.10

Figure 3.5 Load-based routing flow. The application tracks active requests per provider and routes each new request to the provider with the least load. Counters are incremented before the request and decremented after completion (whether successful or failed), ensuring accurate load tracking.

Trace the request lifecycle from provider-load measurement through least-loaded selection, counter updates, completion, and failure-safe cleanup. section 3.5.3

How Designing AI Systems covers it

How deep it goes
Defines least-load selection and counter lifecycle, while stopping short of provider-specific implementation code.
Examples
Uses multiple providers, multiple provider accounts, and rate-limit-induced fallback as operational examples.
What it assumes
Assumes multiple usable providers or provider accounts and awareness of rate limits and response-consistency requirements.
What you can do after
Build it from scratch by tracking per-provider active requests and dispatching each request to the least-loaded provider.
How it connects
Connects to model routing, rate limiting, retries, fallbacks, API gateways, and observability.

Also explained in

Compare how 17 books explain load balancing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.