Load balancing routes each request to the provider with the least current workload instead of concentrating traffic on one provider until failure. This proactively uses aggregate capacity, increases throughput, and reduces rate-limit pressure. section 3.5.3
Implement the routing pattern with a small amount of explicit state:
- Track active requests independently for every provider. section 3.5.3
- Select the provider with the lowest current counter when a request arrives. section 3.5.3
- Increment the selected provider's counter before dispatching the request. section 3.5.3
- Decrement the counter after completion, regardless of success or failure, so routing state remains accurate. section 3.5.3
Use this strategy when multiple providers expose useful capacity, requests have broadly similar latency, or rate limits make concentrating traffic risky. Prefer equivalent provider accounts or keys when response consistency matters: different model families can produce different outputs, which may be unsuitable for schema-sensitive extraction. section 3.5.3
Operational metrics make the decision measurable. Rising latency, frequent rate-limit fallbacks, provider-specific errors, and throughput changes can indicate that traffic should be redistributed rather than sent repeatedly to one provider. section 3.8.3 section 3.10
Figure 3.5 Load-based routing flow. The application tracks active requests per provider and routes each new request to the provider with the least load. Counters are incremented before the request and decremented after completion (whether successful or failed), ensuring accurate load tracking.

Trace the request lifecycle from provider-load measurement through least-loaded selection, counter updates, completion, and failure-safe cleanup. section 3.5.3
















