Llm service

As explained in LLMs in Production

An LLM service is a deployed software service that exposes a large language model for remote requests and response generation.

An LLM service packages a language model behind an API so applications can send requests and receive generated responses on demand. This turns a trained model into a live software component that can serve chatbots, batch jobs, streaming pipelines, or editor integrations. section 6.0 section 6.1

A practical service usually combines the model with an API layer that handles:

• Request preprocessing and retrieval-augmented generation (RAG) before inference. section 6.1
• Batching, rate limiting, and streaming to manage workload and responsiveness. section 6.1
• Deployment on a cloud cluster or an edge device, with explicit trade-offs in latency, efficiency, accuracy, memory, and cost. section 6.0 section 11.0

A small network service on a Raspberry Pi

A model server can expose an LLM on a Raspberry Pi for devices on the same home network. An existing application using the OpenAI Python bindings can target the local service by changing base_url and supplying the server's API key. The compatibility makes integration simple, but the Pi may produce only about five tokens per second, illustrating the gap between a working service and a useful enterprise deployment. section 11.0 section 11.3

Figure 6.1 A basic LLM service. A majority of the logic is handled by the API layer, which will ensure the correct preprocessing of incoming requests is done and serve the actual inference of the request.

The architecture diagram shows the API layer coordinating request preprocessing and model inference, making the service boundary and its main responsibilities explicit. section 6.1

How LLMs in Production covers it

How deep it goes

Explains API architecture, preprocessing, RAG, batching, rate limiting, streaming, deployment, and operational trade-offs.

Examples

Uses a basic API architecture, a VS Code coding copilot, and an OpenAI-compatible Raspberry Pi server as concrete examples.

What it assumes

Assumes trained or fine-tuned models and introduces production deployment, API planning, and operational limitations.

What you can do after

Builds and deploys an LLM API, then improves it with preprocessing, RAG, batching, rate limiting, streaming, and deployment choices.

How it connects

Connects LLM services to APIs, RAG, online inference, Kubernetes, rate limiting, streaming, batch processing, and edge computing.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

Related concepts

• applicationused by · An application is a software program or system that performs tasks or provides functionality for users or other software systems.
• contractuses · An API contract is a specification of the operations, data, and behavioral expectations that govern interactions between software components.
• llama.cppused by · llama.cpp is an open-source C/C++ software project and runtime for running large language models efficiently on local hardware.
• retrieval-augmented generationuses · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
• language modelimplemented by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.