An LLM gateway routes language-model requests across providers with different APIs, pricing, and rate limits. It extends existing platform infrastructure rather than replacing it. section 12.1.2 section 12.1.2
Production gateways add LLM-specific operational controls:
- Route requests across providers and models instead of identical backend servers. section 12.1.2
- Apply token-budget rate limits rather than request-count limits. section 12.1.2
- Use semantic response caching so differently worded questions can share cached results. section 12.1.2
- Fall back between providers when failures such as context-length limits occur. section 12.1.2
Figure 12.2 Traditional ML models and LLM applications coexist in the same cluster, managed by identical infrastructure components. The same Horizontal Pod Auto-Scaler (HPA) that scales your XGBoost recommendation engine also scales your LangChain RAG application. No separate infrastructure is needed—just different workload types sharing proven K8s orchestration.
Place the gateway within shared platform infrastructure, where Kubernetes ingress and autoscaling support both traditional ML and LLM workloads. section 12.1.2
