Chapter 1: Why companies self-host large language models
This introduction contrasts provider-hosted LLM APIs with self-hosting and frames self-hosting as an increasingly accessible infrastructure task.
After this chapter you will be able to
- Self-hosting and provider APIs. Distinguish running an LLM on computers an organization controls from sending requests to a provider that runs the model and charges for answers.
- Self-hosting as infrastructure work. Explain why the section frames running a model as an accessible infrastructure task for platform teams.
- Model Hosting Alternatives. Distinguish a public model API, self-hosting, and a managed cloud model service by who runs the model and where requests can stay.
- Confidentiality Motivation. Explain how the legal firm's chatbot incident illustrates a reason to self-host.
- Prompt privacy and self-hosting. Explain why sensitive workloads may favor a correctly designed self-hosted deployment over a model API.
- End-to-end serving privacy. Explain why privacy must be assessed across the entire serving system rather than just the model machines or weight files.
- API and self-hosted costs. Explain how API and self-hosted cost structures differ as usage grows, including the conditions under which an extra self-hosted request may cost little.
- Serving latency and availability. Explain how owning the serving system changes control over latency and availability compared with using a hosted API.
- Model customization and versions. Describe how self-hosting gives an organization control over model customization and version changes.
- Usage policy and preparedness. Explain how the security-incident example illustrates provider usage-policy restrictions and the value of having a vetted self-hosted model ready.
- Single provider concentration risk. Explain why relying on one external provider for a core business capability can motivate retaining control of model serving.
- Regulatory evidence requirements. Explain how regulation adds evidentiary and processing-location requirements beyond data privacy, and describe the evidence regulators or client auditors can evaluate.
- Self-hosting and compliance limits. Explain how self-hosting can help demonstrate controls without automatically establishing compliance, and contrast it with a hosted option.
- Open-weight model quality. Explain how improvements in open-weight models changed the self-hosting decision.
- Inference engine serving roles. Describe what an inference engine does and why this book uses vLLM.
- Multi-machine operation and reliability. Explain Kubernetes's role in running vLLM across machines and why reliable operation matters beyond model rankings.
- Major model family landscape. Describe several prominent self-hosted model families and how their offerings differ in tasks, deployment range, or design.
- Family lineup structure. Explain how a model family can span sizes and designs, and why operational experience with one member can remain useful across the family.
- Dense and expert architectures. Explain how dense and Mixture of Experts models process tokens differently, including the roles of attention, feed-forward layers, and routing.
- Parameter counts and sizing. Explain how total and active parameter counts affect memory and computation in dense and MoE models.
- Architecture and answer quality. Explain why dense versus MoE architecture alone does not determine answer quality, and how to compare models in practice.
- Model size and memory. Explain how model size affects the memory and hardware needed to host it, using the small-model band as an example.
- Practical model size bands. Distinguish the small, medium, and large model bands by their approximate sizes, typical uses, and hardware demands.
- Model selection and leaderboards. Explain how to use model size and public leaderboards when choosing candidates for a workload.
- Self Hosting and Control. Explain what makes a model self-hosted and what the organization controls.
- Provider API Data Boundary. Distinguish the data path of a third-party model API from that of a self-hosted model.
- Original families and derivatives. Explain how the large open-weight landscape relates to a smaller set of original models, including what fine-tunes and quantized versions are.
- Open-weight and open-source distinction. Distinguish open-weight models from open-source AI, including why permissively licensed weights alone do not establish that the whole system is open source.
- Model license review. Explain why a team should check the license on the exact model it plans to use, and describe how license conditions affect enterprise deployment.
- Upstream model lineage. Explain why a fine-tune or distilled model requires review of its upstream lineage rather than only the final repository's license.
- Layered inference platform. Explain what an inference platform includes and why its layered architecture can be adopted incrementally.
- Hardware and model artifacts. Describe how hardware constrains deployment and what model artifacts must be managed.
- Inference engine responsibilities. Describe what the inference engine does between the model files and the hardware.
- Serving API compatibility. Explain the serving API's application-facing role and the benefit and limits of OpenAI-compatible requests.
- Gateway routing and protection. Explain how the gateway makes a service available across an organization and why its logs need protection.
- Self Hosting Decision Framework. Explain how the five forces and the signals in this section guide a self-hosting decision.
- Usage and Prototyping Signals. Explain why low or irregular usage and early prototyping can favor a hosted API.
- Model Capability and Ownership. Explain why model capability requirements and lack of platform ownership can favor a hosted API.
- Lab hardware and access. Describe how to access the early labs and how hardware needs change later in the book.
- Software and model access. Describe the software skills and account needed for the book, and explain why tool versions are pinned.
Section by section
Introduction
This introduction contrasts provider-hosted LLM APIs with self-hosting and frames self-hosting as an increasingly accessible infrastructure task.
- Self-hosting and provider APIs: Distinguish running an LLM on computers an organization controls from sending requests to a provider that runs the model and charges for answers.
Why it matters: This distinction establishes what changes when an organization takes responsibility for running the model. - Self-hosting as infrastructure work: Explain why the section frames running a model as an accessible infrastructure task for platform teams.
Why it matters: It connects LLM self-hosting to the service operations skills platform teams already use.
Key terms: self-hosting, LLM, open source, infrastructure work, predictable cost
1.1 Understanding the shift from API calls to owned infrastructure
This section contrasts self-hosted models with hosted services and introduces the confidentiality need that can motivate the shift.
- Model Hosting Alternatives: Distinguish a public model API, self-hosting, and a managed cloud model service by who runs the model and where requests can stay.
Why it matters: These alternatives determine which systems handle your data and who operates the model. - Confidentiality Motivation: Explain how the legal firm's chatbot incident illustrates a reason to self-host.
Why it matters: A model's useful answer does not by itself resolve concerns about where confidential input goes. - LLM Text Capabilities: Describe an LLM's basic text capability and the scope of text examples in this book.
Why it matters: This establishes what the model does and what kinds of input the book's examples cover.
Key terms: API, self-hosting, infrastructure, managed model services, confidentiality, data loss prevention, public AI chatbot, LLM, multimodal, text
1.1.1 Force 1: Data privacy and confidentiality
This section explains why sensitive prompts may motivate self-hosting and why privacy depends on the entire serving system’s data path.
- Prompt privacy and self-hosting: Explain why sensitive workloads may favor a correctly designed self-hosted deployment over a model API.
Why it matters: Where prompts travel can determine whether a deployment is suitable for sensitive data. - End-to-end serving privacy: Explain why privacy must be assessed across the entire serving system rather than just the model machines or weight files.
Why it matters: A private model machine can still be part of a system that exposes request data.
Key terms: prompts, model API, self-hosted deployment, data path, external monitoring service, application logs, serving system
1.1.2 Force 2: Cost at scaleCost at scale
This section contrasts per-token API pricing with the capacity-based costs of self-hosting as usage grows.
- API and self-hosted costs: Explain how API and self-hosted cost structures differ as usage grows, including the conditions under which an extra self-hosted request may cost little.
Why it matters: The different cost structures help explain why an inexpensive API prototype can become costly at production scale. - Organization-specific cost comparison: Describe why deciding whether self-hosting saves money requires a comparison using the organization's actual costs.
Why it matters: Using actual prices and measurements avoids assuming that self-hosting automatically saves money.
Key terms: tokens, per-token pricing, GPU capacity, marginal infrastructure cost, subscriptions, committed-use discounts, measured numbers
1.1.3 Force 3: Control
This section explains how self-hosting gives organizations more control over model service behavior, customization, versions, and usage restrictions.
- Serving latency and availability: Explain how owning the serving system changes control over latency and availability compared with using a hosted API.
Why it matters: Control over response time and availability matters when an application cannot simply accept a provider’s service behavior. - Model customization and versions: Describe how self-hosting gives an organization control over model customization and version changes.
Why it matters: Control over customization and version changes helps an organization decide what model it runs and when it changes. - Usage policy and preparedness: Explain how the security-incident example illustrates provider usage-policy restrictions and the value of having a vetted self-hosted model ready.
Why it matters: A prepared self-hosted option can support legitimate work that hosted usage policies block while keeping sensitive incident data local.
Key terms: latency, availability, serving system, fine-tuning, model version, usage policy, safety guardrails, forensic analysis, open-weight model
1.1.4 Force 4: Vendor concentration risk
This section explains why dependence on a single external model provider can make organizations want to retain control of model serving.
- Single provider concentration risk: Explain why relying on one external provider for a core business capability can motivate retaining control of model serving.
Why it matters: This risk can make owning model-serving capability attractive even when an external provider works well today.
Key terms: vendor concentration risk, model serving
1.1.5 Force 5: Regulation
This section explains how regulatory obligations motivate self-hosting while distinguishing demonstrable controls from compliance itself.
- Regulatory evidence requirements: Explain how regulation adds evidentiary and processing-location requirements beyond data privacy, and describe the evidence regulators or client auditors can evaluate.
Why it matters: Handling data carefully is not enough if you cannot provide evidence that others can evaluate. - Self-hosting and compliance limits: Explain how self-hosting can help demonstrate controls without automatically establishing compliance, and contrast it with a hosted option.
Why it matters: You can evaluate self-hosting as a way to support compliance without mistaking it for compliance itself. - Regional regulatory context: Distinguish the section's examples of EU and US regulatory pressures.
Why it matters: The applicable regulatory pressure depends on the jurisdiction and context of the workload.
Key terms: regulation, data location, access controls, auditability, self-hosting, compliance, compliance agreements, trust model, GDPR, EU AI Act
1.1.6 Why self-hosting is possible now
This section explains why self-hosting LLMs has become practical and why existing deployments may need to migrate to supported inference engines.
- Open-weight model quality: Explain how improvements in open-weight models changed the self-hosting decision.
Why it matters: Model quality now makes self-hosting a practical choice for more workloads. - Inference engine serving roles: Describe what an inference engine does and why this book uses vLLM.
Why it matters: The inference engine turns model files into a service that users can request answers from. - Multi-machine operation and reliability: Explain Kubernetes's role in running vLLM across machines and why reliable operation matters beyond model rankings.
Why it matters: A capable model still needs reliable infrastructure to serve real users. - TGI migration pressure: Explain why organizations using Text Generation Inference need to consider a replacement.
Why it matters: An existing inference deployment may need to move to software that remains supported.
Key terms: open-weight model, weights, inference engine, vLLM, GPU, API, Kubernetes, production, Text Generation Inference, TGI
1.1.7 The major families
This section maps prominent open-weight model families and explains why operators should understand family lineups and track changing releases.
- Major model family landscape: Describe several prominent self-hosted model families and how their offerings differ in tasks, deployment range, or design.
Why it matters: Recognizing the range of families helps you navigate the choices available for self-hosting. - Family lineup structure: Explain how a model family can span sizes and designs, and why operational experience with one member can remain useful across the family.
Why it matters: Thinking in families helps you reuse operational experience rather than starting over with each model. - Model version change: Describe how operators should respond to the rapid evolution of model generations.
Why it matters: Checking new releases helps you avoid treating an evolving model as a fixed dependency. - Family license differences: Describe examples of how terms for accessing or using models differ across the listed families.
Why it matters: License terms are part of distinguishing models that might otherwise look like similar open-weight choices.
Key terms: Llama, Muse, Qwen, Mistral, DeepSeek, Kimi, model family, lineup, dense, Mixture of Experts
1.1.8 The axis that matters most: dense versus Mixture of Experts
This section contrasts dense and Mixture of Experts models and explains how their architectures affect inference sizing and model comparisons.
- Dense and expert architectures: Explain how dense and Mixture of Experts models process tokens differently, including the roles of attention, feed-forward layers, and routing.
Why it matters: This distinction explains why the two designs can have different memory and computation needs. - Parameter counts and sizing: Explain how total and active parameter counts affect memory and computation in dense and MoE models.
Why it matters: You need both counts to estimate the memory and computation an MoE deployment requires. - Architecture and answer quality: Explain why dense versus MoE architecture alone does not determine answer quality, and how to compare models in practice.
Why it matters: Architecture helps estimate resources, but it is not a substitute for comparing answer quality on your own tasks.
Key terms: dense, Mixture of Experts, MoE, attention layer, feed-forward layer, experts, parameters, total parameter count, active parameter count, memory
1.1.9 Sizes and what they mean in practice
This section relates model size bands to hardware needs and practical capabilities, and recommends selecting the smallest model that meets the workload’s quality bar.
- Model size and memory: Explain how model size affects the memory and hardware needed to host it, using the small-model band as an example.
Why it matters: Model size helps you estimate the hardware needed before anyone uses the service. - Practical model size bands: Distinguish the small, medium, and large model bands by their approximate sizes, typical uses, and hardware demands.
Why it matters: The bands give you a starting point for matching workload difficulty to hosting resources. - Model selection and leaderboards: Explain how to use model size and public leaderboards when choosing candidates for a workload.
Why it matters: This keeps model selection focused on your workload rather than on size or rankings alone.
Key terms: parameters, GPU memory, weights, 16-bit precision, quantized copy, small, medium, large, dense, MoE
1.2 What "self-hosting" actually means
This section defines self-hosting by organizational control, contrasts its data boundary with a provider API, and notes the exception introduced by external tools.
- Self Hosting and Control: Explain what makes a model self-hosted and what the organization controls.
Why it matters: Control, rather than the location or ownership of the hardware, defines the self-hosting boundary. - Provider API Data Boundary: Distinguish the data path of a third-party model API from that of a self-hosted model.
Why it matters: Using an open-weight model does not by itself keep requests within your controlled systems. - External Tool Data Exposure: Explain how external tool calls can change the data boundary of a self-hosted model.
Why it matters: A self-hosted model can still send context outside your systems through its tools.
Key terms: self-hosting, control, neoclouds, open-weight model, third-party API, data path, tool calling, external services, context
1.3 Surveying the open-weight model landscape
This section maps the open-weight landscape as a smaller set of original model families surrounded by many derivatives and suggests a way to browse them.
- Original families and derivatives: Explain how the large open-weight landscape relates to a smaller set of original models, including what fine-tunes and quantized versions are.
Why it matters: Recognizing derivatives helps make the seemingly enormous model landscape easier to navigate. - Model browsing and popularity: Describe how to explore models on the Hugging Face Hub and the limitation of popularity as a guide.
Why it matters: Popularity can help you start exploring models, but it is not evidence of production use.
Key terms: open-weight, fine-tunes, quantized versions, derivatives, Hugging Face Hub, downloads, trending, popularity
1.4 Distinguishing open-weight from open-source, and why licenses matter
This section distinguishes open-weight from open-source AI and explains why model licenses and upstream lineage must be reviewed before enterprise deployment.
- Open-weight and open-source distinction: Distinguish open-weight models from open-source AI, including why permissively licensed weights alone do not establish that the whole system is open source.
Why it matters: Downloadable weights do not tell you either what uses are allowed or whether the full system is open source. - Model license review: Explain why a team should check the license on the exact model it plans to use, and describe how license conditions affect enterprise deployment.
Why it matters: A license can rule out a model or impose obligations before you even test its quality. - Upstream model lineage: Explain why a fine-tune or distilled model requires review of its upstream lineage rather than only the final repository's license.
Why it matters: A permissive-looking final repository may not reveal all the terms relevant to a derived model.
Key terms: open-weight, open-source AI, trained parameters, permissive license, permissive licenses, custom licenses, model card, acceptable-use policies, fine-tune, distillation
1.5 Sketching the anatomy of an inference platform
This section sketches the layers of an LLM inference platform and the role each plays in turning a model into a usable service.
- Layered inference platform: Explain what an inference platform includes and why its layered architecture can be adopted incrementally.
Why it matters: The model alone is not the service your users operate against. - Hardware and model artifacts: Describe how hardware constrains deployment and what model artifacts must be managed.
Why it matters: Hardware determines what can run, while the right files must be available and kept consistent. - Inference engine responsibilities: Describe what the inference engine does between the model files and the hardware.
Why it matters: The engine makes the model executable and handles shared hardware resources. - Serving API compatibility: Explain the serving API's application-facing role and the benefit and limits of OpenAI-compatible requests.
Why it matters: A familiar API can make moving an application to a self-hosted service easier. - Inference workload orchestration: Describe orchestration and the inference-specific handling Kubernetes needs.
Why it matters: Inference workloads need operational settings that account for GPUs and slow startup. - Gateway routing and protection: Explain how the gateway makes a service available across an organization and why its logs need protection.
Why it matters: A shared entry point enables organizational controls but also concentrates sensitive information.
Key terms: inference platform, observability, governance, GPUs, model artifacts, weight files, tokenizer, adapters, inference engine, GPU memory
1.6 Deciding when self-hosting is the wrong choice
This section explains when a hosted API is preferable to self-hosting and how a hybrid approach can serve different workloads.
- Self Hosting Decision Framework: Explain how the five forces and the signals in this section guide a self-hosting decision.
Why it matters: This framework helps you avoid committing to infrastructure without a strong reason. - Usage and Prototyping Signals: Explain why low or irregular usage and early prototyping can favor a hosted API.
Why it matters: Usage patterns and product maturity can make an API the more practical starting point. - Model Capability and Ownership: Explain why model capability requirements and lack of platform ownership can favor a hosted API.
Why it matters: A hosting decision must account for both the capability the workload needs and who will keep the service running. - Hybrid Workload Routing: Describe the hybrid hosting option given in the section.
Why it matters: You can choose hosting separately for workloads rather than making an all-or-nothing decision.
Key terms: data privacy, cost at scale, control, vendor concentration risk, regulation, per-token pricing, low or irregular volume, prototyping, OpenAI-compatible API, strongest models
1.7 What you’ll need for this book
This section outlines the accounts, skills, hardware, and software needed to work through the book.
- Lab hardware and access: Describe how to access the early labs and how hardware needs change later in the book.
Why it matters: You can start for free while planning for the infrastructure later labs require. - Software and model access: Describe the software skills and account needed for the book, and explain why tool versions are pinned.
Why it matters: These prerequisites and version choices help you run the examples and obtain models. - Book code repository: State where the book's code and lab notebooks are available.
Why it matters: The repository lets you find the examples and complete notebooks without recreating them.
Key terms: Kaggle, free tier, GPUs, Kubernetes cluster, Python, command line, Hugging Face, pinned versions, repository, listings
1.8 Summary
Concepts explained in this chapter
- self-hostingA deployment in which an organization runs and manages a model within infrastructure it controls.
- open-weight modelA model whose trained parameters are made available for others to download or use, without necessarily making its training data or code open.
- inference engineAn inference engine is software that loads a trained machine-learning model and computes outputs for input requests.
- mixture of expertsA mixture of experts model is a model whose expert components are selectively activated for each token by a routing component.
- dense modelA dense model is a neural network in which the model’s parameters participate in computation for each input, rather than selecting a sparse subset.
- inference platformA system surrounding a machine-learning model that serves it through an API and provides operational functions such as routing, protection, and monitoring.
- vLLMvLLM is an open-source inference and serving engine for large language models, designed to serve many users efficiently.
- large language modelA machine learning model trained on large text collections to understand and generate language.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
