liveBook · Manning

Chapter 1: Why companies self-host large language models

What you will learn in chapter 1 of Self-Hosting LLMs at Scale

This introduction contrasts provider-hosted LLM APIs with self-hosting and frames self-hosting as an increasingly accessible infrastructure task.

After this chapter you will be able to

Section by section

Introduction

This introduction contrasts provider-hosted LLM APIs with self-hosting and frames self-hosting as an increasingly accessible infrastructure task.

Key terms: self-hosting, LLM, open source, infrastructure work, predictable cost

1.1 Understanding the shift from API calls to owned infrastructure

This section contrasts self-hosted models with hosted services and introduces the confidentiality need that can motivate the shift.

Key terms: API, self-hosting, infrastructure, managed model services, confidentiality, data loss prevention, public AI chatbot, LLM, multimodal, text

1.1.1 Force 1: Data privacy and confidentiality

This section explains why sensitive prompts may motivate self-hosting and why privacy depends on the entire serving system’s data path.

Key terms: prompts, model API, self-hosted deployment, data path, external monitoring service, application logs, serving system

1.1.2 Force 2: Cost at scaleCost at scale

This section contrasts per-token API pricing with the capacity-based costs of self-hosting as usage grows.

Key terms: tokens, per-token pricing, GPU capacity, marginal infrastructure cost, subscriptions, committed-use discounts, measured numbers

1.1.3 Force 3: Control

This section explains how self-hosting gives organizations more control over model service behavior, customization, versions, and usage restrictions.

Key terms: latency, availability, serving system, fine-tuning, model version, usage policy, safety guardrails, forensic analysis, open-weight model

1.1.4 Force 4: Vendor concentration risk

This section explains why dependence on a single external model provider can make organizations want to retain control of model serving.

Key terms: vendor concentration risk, model serving

1.1.5 Force 5: Regulation

This section explains how regulatory obligations motivate self-hosting while distinguishing demonstrable controls from compliance itself.

Key terms: regulation, data location, access controls, auditability, self-hosting, compliance, compliance agreements, trust model, GDPR, EU AI Act

1.1.6 Why self-hosting is possible now

This section explains why self-hosting LLMs has become practical and why existing deployments may need to migrate to supported inference engines.

Key terms: open-weight model, weights, inference engine, vLLM, GPU, API, Kubernetes, production, Text Generation Inference, TGI

1.1.7 The major families

This section maps prominent open-weight model families and explains why operators should understand family lineups and track changing releases.

Key terms: Llama, Muse, Qwen, Mistral, DeepSeek, Kimi, model family, lineup, dense, Mixture of Experts

1.1.8 The axis that matters most: dense versus Mixture of Experts

This section contrasts dense and Mixture of Experts models and explains how their architectures affect inference sizing and model comparisons.

Key terms: dense, Mixture of Experts, MoE, attention layer, feed-forward layer, experts, parameters, total parameter count, active parameter count, memory

1.1.9 Sizes and what they mean in practice

This section relates model size bands to hardware needs and practical capabilities, and recommends selecting the smallest model that meets the workload’s quality bar.

Key terms: parameters, GPU memory, weights, 16-bit precision, quantized copy, small, medium, large, dense, MoE

1.2 What "self-hosting" actually means

This section defines self-hosting by organizational control, contrasts its data boundary with a provider API, and notes the exception introduced by external tools.

Key terms: self-hosting, control, neoclouds, open-weight model, third-party API, data path, tool calling, external services, context

1.3 Surveying the open-weight model landscape

This section maps the open-weight landscape as a smaller set of original model families surrounded by many derivatives and suggests a way to browse them.

Key terms: open-weight, fine-tunes, quantized versions, derivatives, Hugging Face Hub, downloads, trending, popularity

1.4 Distinguishing open-weight from open-source, and why licenses matter

This section distinguishes open-weight from open-source AI and explains why model licenses and upstream lineage must be reviewed before enterprise deployment.

Key terms: open-weight, open-source AI, trained parameters, permissive license, permissive licenses, custom licenses, model card, acceptable-use policies, fine-tune, distillation

1.5 Sketching the anatomy of an inference platform

This section sketches the layers of an LLM inference platform and the role each plays in turning a model into a usable service.

Key terms: inference platform, observability, governance, GPUs, model artifacts, weight files, tokenizer, adapters, inference engine, GPU memory

1.6 Deciding when self-hosting is the wrong choice

This section explains when a hosted API is preferable to self-hosting and how a hybrid approach can serve different workloads.

Key terms: data privacy, cost at scale, control, vendor concentration risk, regulation, per-token pricing, low or irregular volume, prototyping, OpenAI-compatible API, strongest models

1.7 What you’ll need for this book

This section outlines the accounts, skills, hardware, and software needed to work through the book.

Key terms: Kaggle, free tier, GPUs, Kubernetes cluster, Python, command line, Hugging Face, pinned versions, repository, listings

1.8 Summary

Concepts explained in this chapter

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.