When an organization needs to serve a language model to many users on systems it controls, vLLM provides an open-source inference and serving engine. An inference engine loads a trained model and answers requests efficiently. section 1.0 section 1.1.6
vLLM processes requests from multiple users to keep the GPU busy, manages temporary conversation memory, and provides an API compatible with common commercial services. Unlike vLLM’s multi-user focus, llama.cpp and tools built on it, such as Ollama and LM Studio, are popular for running a model on a single machine. section 1.1.6
When one machine is no longer enough, Kubernetes can manage vLLM across several connected machines. Reliable operation still requires attention to model memory, load time, machine failures, and slow responses. section 1.1.6
Figure 1.1 Kubernetes runs programs across several connected machines. Its control plane decides which machine runs each copy of vLLM, restarts copies that fail, and adds machines and copies as traffic grows.

See how Kubernetes places and restarts vLLM copies and expands capacity as traffic grows. section 1.1.6



