A team handling confidential documents may need a language model without sending prompts to an external model provider. Self-hosting means running a downloaded model on infrastructure the organization controls. That infrastructure can be a company data center, rented cloud GPU servers, or dedicated capacity from a GPU provider; it need not be in the organization’s building. section 1.1 section 1.2
Figure 1.2 Two data-flow diagrams side by side. Left: application, then public internet, then provider API on provider infrastructure, with data leaving the systems the company controls. Right: application, then company gateway, then inference engine on company-controlled GPUs, with data staying inside the systems the company controls.

Compare the data path through a provider API with the path through a company-controlled inference service. section 1.2
Control comes with ownership. The team chooses the model and version, controls access, and operates the service. A model accessed through a third-party API is not self-hosted just because its weights are available to download. Managed model services are an intermediate option: requests may stay inside a cloud account, but the provider runs the model and charges per token. section 1.2 section 1.1
Evaluate the decision against the workload and the team’s operating capacity:
- Privacy and compliance: check the entire data path, including application logs, monitoring exports, and external tools—not just where the model runs. section 1.1.1 section 1.2
- Cost: compare actual API pricing with GPU capacity, electricity, maintenance, and staffing. Spare capacity can make another request inexpensive, but it still consumes resources. section 1.1.2
- Control and risk: weigh model-version and access control, provider usage policies, vendor concentration, and regulatory requirements. section 1.1.3 section 1.8
- Operational fit: retain an API when volume is low or irregular, prototyping is early, the strongest commercial model is essential, or no team can own incidents and upgrades. section 1.6
- Model selection: check the license before quality testing; open-weight does not necessarily mean open-source. section 1.8
A private GPU is not enough
A deployment had dedicated computers, controlled access, and no public internet address. Tracing a request revealed that monitoring metadata went to an external service and application logs could capture user input. Privacy required fixing those paths as well as securing the model server. section 1.1.1
A serving platform combines hardware, model artifacts, an inference engine, an API, orchestration, a gateway, and observability and governance. For example, vLLM serves concurrent users and manages conversation memory; Kubernetes can manage copies across machines when one machine is no longer enough. Plan for monitoring, failures, and response-time degradation as part of the service. section 1.8 section 1.1.6








