chapter one

1 Why companies self-host large language models

 

This chapter covers

  • The why of self-hosting
  • The five forces
  • The landscape of open-weight models
  • The layers of self-hosting
  • When self-hosting is wrong

Most companies connect to a large language model (LLM) by sending requests over the internet to a provider that runs the model and charges for every answer. This book is about an alternate path: running such a model yourself, on computers your organization controls.

Running a model yourself used to require machine learning specialists and a large hardware budget. It no longer does: the models are downloadable, the engines are open source, and the work is infrastructure work. The engineering problem is one that platform teams already know: run a service the business depends on, on systems the organization controls, with predictable cost and a clear record of where the data goes. This book is written for the platform engineers, DevOps and backend developers, and technical leads asked to turn a language model into a reliable internal service. No prior experience with language models or GPUs is expected; comfort with Python and the command line is all you need to bring. You will learn on hardware that costs nothing: the labs run on Kaggle's free tier with two Tesla T4 GPUs.

1.1 Understanding the shift from API calls to owned infrastructure

1.1.1 Force 1: Data privacy and confidentiality

1.1.2 Force 2: Cost at scaleCost at scale

1.1.3 Force 3: Control

1.1.4 Force 4: Vendor concentration risk

1.1.5 Force 5: Regulation

1.1.6 Why self-hosting is possible now

1.2 What "self-hosting" actually means

1.3 Surveying the open-weight model landscape

1.1.7 The major families

1.1.8 The axis that matters most: dense versus Mixture of Experts

1.1.9 Sizes and what they mean in practice

1.4 Distinguishing open-weight from open-source, and why licenses matter

1.5 Sketching the anatomy of an inference platform

1.6 Deciding when self-hosting is the wrong choice

1.7 What you’ll need for this book

1.8 Summary