When an organization self-hosts a language model, it needs software to turn model files into answers. An inference engine loads the trained weights, runs the model, and handles requests efficiently. It is one part of an inference platform, not the whole service around it. section 1.1.6 section 1.5
The engine manages GPU memory and groups concurrent requests to keep the GPU busy. If a model exceeds one GPU or server, it can split the model across multiple devices or machines. The surrounding platform adds capabilities such as request routing, protection, monitoring, and orchestration. section 1.1.6 section 1.5
Figure 1.5 A single request traveling through the platform. (1) The user's application sends the question to the serving API, the platform's single visible address. (2) The API server prepares the request and converts the text into tokens, the numeric units the model reads. (3) The inference engine schedules the request onto the GPU alongside requests from other users. (4) The GPU runs the model: it first processes the whole prompt, then produces the answer one token at a time. (5) Each token streams back through the same path, so the user sees the answer appear word by word while it is still being generated. The gateway, orchestration, and monitoring layers surround this path in production, routing the request to the right server and recording what happened.

Follow a request from the serving API through engine scheduling and GPU execution, then back as streamed answer tokens. section 1.5
Choose an engine for the workload
For a model running on one machine, llama.cpp and tools built on it, such as Ollama and LM Studio, are popular choices. vLLM is designed to serve many users concurrently and provides an API compatible with common commercial services. Basic OpenAI-style endpoints can often migrate with minimal configuration changes, but advanced features still need compatibility testing. section 1.1.6 section 1.5




