liveBook · Manning

Inference platform

As explained in Self-Hosting LLMs at Scale

A system surrounding a machine-learning model that serves it through an API and provides operational functions such as routing, protection, and monitoring.

A model alone is not a service. An inference platform surrounds a model with the components needed to run it, accept API requests, and route, protect, and monitor traffic. In a self-hosted deployment, the organization controls the infrastructure and must decide who will operate the platform. section 1.5 section 1.8

The platform has several layers, from hardware to operational controls:

  • Hardware: GPUs, their memory, and their connections constrain which models can run and how quickly. section 1.5
  • Model artifacts: weight files, a tokenizer that matches the weights, and optional adapters must be stored, cached, and managed through changes and rollbacks. A tokenizer converts text into the numeric tokens a model processes. section 1.5
  • Inference engine: loads weights, manages GPU memory, groups concurrent requests, and can split a model across GPUs or servers. section 1.5
  • Serving API: gives applications an address for sending requests. An OpenAI-compatible API can ease migration for basic endpoints, but advanced features still need compatibility testing. section 1.5
  • Orchestration and gateway: orchestration places, restarts, and scales programs; the gateway routes requests and enforces limits. section 1.5
  • Observability and governance: monitoring, cost tracking, and governance span the other layers. section 1.5
Figure 1.4 The inference platform, shown as layers from bottom to top: (1) GPU hardware and memory; (2) the model files; (3) the inference engine that runs the model; (4) the API that receives requests; (5) software that places, restarts, and scales programs across computers; (6) a gateway that routes requests and enforces limits; and (7) monitoring, cost tracking, and governance across all layers. The solid arrow shows requests flowing down from the gateway toward the hardware; the dashed arrow shows answers returning. The orchestration layer and the observability plane manage and observe that path rather than carry it.

Use the layered view to distinguish the request path from orchestration and observability, which manage and observe that path. section 1.5

A request through the platform

An application sends a question to the serving API. The server prepares it and converts the text into tokens; the engine schedules it on a GPU alongside other requests. The GPU processes the prompt, then generates the answer one token at a time. Tokens stream back as they are produced, while surrounding gateway and monitoring layers route and record the request. section 1.5

Platform choices should follow the reason for self-hosting, whether privacy, cost at scale, control, vendor concentration risk, or regulation. Self-hosting may be a poor fit for low volume or early prototyping, when the strongest commercial models are essential, or when no team will own the platform. section 1.8

How Self-Hosting LLMs at Scale covers it

How deep it goes
Defines the layers and traces a request through them, while deferring detailed inference-engine treatment.
Examples
Traces an application question from API tokenization through GPU scheduling to streamed answer tokens.
What you can do after
Evaluate self-hosting needs and identify the layers and ownership a deployment requires.
How it connects
Connects inference platforms to GPU hardware, model artifacts, inference engines, APIs, orchestration, gateways, observability, and governance.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Self-Hosting LLMs at Scale. The section links point to the parts of the book it draws on.