A model alone is not a service. An inference platform surrounds a model with the components needed to run it, accept API requests, and route, protect, and monitor traffic. In a self-hosted deployment, the organization controls the infrastructure and must decide who will operate the platform. section 1.5 section 1.8
The platform has several layers, from hardware to operational controls:
- Hardware: GPUs, their memory, and their connections constrain which models can run and how quickly. section 1.5
- Model artifacts: weight files, a tokenizer that matches the weights, and optional adapters must be stored, cached, and managed through changes and rollbacks. A tokenizer converts text into the numeric tokens a model processes. section 1.5
- Inference engine: loads weights, manages GPU memory, groups concurrent requests, and can split a model across GPUs or servers. section 1.5
- Serving API: gives applications an address for sending requests. An OpenAI-compatible API can ease migration for basic endpoints, but advanced features still need compatibility testing. section 1.5
- Orchestration and gateway: orchestration places, restarts, and scales programs; the gateway routes requests and enforces limits. section 1.5
- Observability and governance: monitoring, cost tracking, and governance span the other layers. section 1.5
Figure 1.4 The inference platform, shown as layers from bottom to top: (1) GPU hardware and memory; (2) the model files; (3) the inference engine that runs the model; (4) the API that receives requests; (5) software that places, restarts, and scales programs across computers; (6) a gateway that routes requests and enforces limits; and (7) monitoring, cost tracking, and governance across all layers. The solid arrow shows requests flowing down from the gateway toward the hardware; the dashed arrow shows answers returning. The orchestration layer and the observability plane manage and observe that path rather than carry it.

Use the layered view to distinguish the request path from orchestration and observability, which manage and observe that path. section 1.5
A request through the platform
An application sends a question to the serving API. The server prepares it and converts the text into tokens; the engine schedules it on a GPU alongside other requests. The GPU processes the prompt, then generates the answer one token at a time. Tokens stream back as they are produced, while surrounding gateway and monitoring layers route and record the request. section 1.5
Platform choices should follow the reason for self-hosting, whether privacy, cost at scale, control, vendor concentration risk, or regulation. Self-hosting may be a poor fit for low volume or early prototyping, when the strongest commercial models are essential, or when no team will own the platform. section 1.8
