liveBook · Manning

Inference engine

As explained in Self-Hosting LLMs at Scale

An inference engine is software that loads a trained machine-learning model and computes outputs for input requests.

When an organization self-hosts a language model, it needs software to turn model files into answers. An inference engine loads the trained weights, runs the model, and handles requests efficiently. It is one part of an inference platform, not the whole service around it. section 1.1.6 section 1.5

The engine manages GPU memory and groups concurrent requests to keep the GPU busy. If a model exceeds one GPU or server, it can split the model across multiple devices or machines. The surrounding platform adds capabilities such as request routing, protection, monitoring, and orchestration. section 1.1.6 section 1.5

Figure 1.5 A single request traveling through the platform. (1) The user's application sends the question to the serving API, the platform's single visible address. (2) The API server prepares the request and converts the text into tokens, the numeric units the model reads. (3) The inference engine schedules the request onto the GPU alongside requests from other users. (4) The GPU runs the model: it first processes the whole prompt, then produces the answer one token at a time. (5) Each token streams back through the same path, so the user sees the answer appear word by word while it is still being generated. The gateway, orchestration, and monitoring layers surround this path in production, routing the request to the right server and recording what happened.

Follow a request from the serving API through engine scheduling and GPU execution, then back as streamed answer tokens. section 1.5

Choose an engine for the workload

For a model running on one machine, llama.cpp and tools built on it, such as Ollama and LM Studio, are popular choices. vLLM is designed to serve many users concurrently and provides an API compatible with common commercial services. Basic OpenAI-style endpoints can often migrate with minimal configuration changes, but advanced features still need compatibility testing. section 1.1.6 section 1.5

How Self-Hosting LLMs at Scale covers it

How deep it goes
Defines engine responsibilities and sketches request flow, while deferring detailed vLLM internals.
Examples
Compares vLLM for concurrent users with llama.cpp, Ollama, and LM Studio for single-machine use.
What you can do after
Evaluate and understand engine choices against single-machine and concurrent-user workloads.
How it connects
Connects inference engines to model weights, GPU memory, serving APIs, orchestration, and observability.

Also explained in

Compare how 5 books explain inference engine →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Self-Hosting LLMs at Scale. The section links point to the parts of the book it draws on.