liveBook · Manning

Inference graph

As explained in LLMs in Production

An inference graph is a graph-based deployment pattern that connects models and services into a coordinated inference workflow.

An inference graph coordinates multiple models and services during inference instead of packaging them as one opaque service. This matters when a workflow combines an encoder, language model, classifier, and API, or when components need independent scaling and updates. section 6.2.4

Separating components creates practical deployment benefits. An encoder can be reused for preprocessing and VectorDB population, while the API can change without redeploying the language model. Independent model APIs also allow different component counts, such as one encoder for every two or three language-model instances. section 6.2.4

Seldon deployment

A Seldon Deployment extends Kubernetes with an orchestrator that runs graph components in order. A representative configuration connects an encoder, combiner, language model, and classifier through REST model endpoints.

How LLMs in Production covers it

How deep it goes
Explains sequential, ensemble, and routing structures plus deployment benefits, then illustrates orchestration with Seldon configuration.
Examples
Uses embedding preprocessing, VectorDB population, multi-model coordination, and a Seldon encoder–LLM–classifier deployment.
What it assumes
Assumes familiarity with models and services while demonstrating component APIs, model composition, and Kubernetes-based orchestration.
What you can do after
Operate/configure coordinated model workflows and select graph structures for preprocessing, ensemble combination, or traffic routing.
How it connects
Connects encoders, language models, classifiers, APIs, VectorDBs, Kubernetes, REST endpoints, and Seldon.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.