1 Why companies self-host large language models
This chapter covers
- The why of self-hosting
- The five forces
- The landscape of open-weight models
- The layers of self-hosting
- When self-hosting is wrong
Most companies connect to a large language model (LLM) by sending requests over the internet to a provider that runs the model and charges for every answer. This book is about an alternate path: running such a model yourself, on computers your organization controls.
Running a model yourself used to require machine learning specialists and a large hardware budget. It no longer does: the models are downloadable, the engines are open source, and the work is infrastructure work. The engineering problem is one that platform teams already know: run a service the business depends on, on systems the organization controls, with predictable cost and a clear record of where the data goes. This book is written for the platform engineers, DevOps and backend developers, and technical leads asked to turn a language model into a reliable internal service. No prior experience with language models or GPUs is expected; comfort with Python and the command line is all you need to bring. You will learn on hardware that costs nothing: the labs run on Kaggle's free tier with two Tesla T4 GPUs.