What will I learn in chapter 6 of LLMs in Production?

This introduction defines production as putting an LLM to work for real users and previews the chapter’s path from serving and cloud deployment to production challenges and edge devices. After reading it you should be able to: Explain that production means integrating a model into a live environment where it performs tasks or serves end users, making it available for real-world use rather than leaving it as an unused project. Explain why the team should plan an LLM API architecture before deploying a trained model: to define functionality, anticipate integration challenges, arrange resources, and set development priorities. Describe how a basic LLM service lets users interact with a model on demand and can serve batch or streaming processes without embedding LLM logic in each pipeline. Explain that service communication adds latency, but the abstraction is often worth it because LLMs are already relatively slow. Explain how compiling lowers a model from a development framework through an intermediate representation to code optimized for deployment hardware, and why this can improve inference speed and cost. Trace the section’s TensorRT example from loading a model and example input through TorchScript tracing, TensorRT compilation, and reloading for inference; explain why target hardware and model support matter. Distinguish the ONNX model format from ONNX Runtime, and describe how Hugging Face Optimum exports and optimizes a model as an alternative to TensorRT. Explain why storing an LLM in an object store can slow service startup, despite its storage and reliability benefits. Describe how the choice of storage strategy depends on the model, infrastructure, system requirements, and traffic. Describe serving a model from a mounted file-based volume rather than an object store, including its startup benefit and the coordination, reliability, and cost tradeoffs.

What does section 6.1, Creating an LLM service with RAG and more, cover?

Introduces why an LLM API should be planned before deployment, what a basic service enables, and the service features the section will examine. Explain why the team should plan an LLM API architecture before deploying a trained model: to define functionality, anticipate integration challenges, arrange resources, and set development priorities. Describe how a basic LLM service lets users interact with a model on demand and can serve batch or streaming processes without embedding LLM logic in each pipeline. Explain that service communication adds latency, but the abstraction is often worth it because LLMs are already relatively slow. Describe the basic service diagram as a simplified view that hides work in the API and preprocessing layers, including batching, rate limiting, streaming, and retrieval-augmented generation. State that the section first turns to preparing the model for online inference before working through these service features.

What does section 6.1.1, Model compilation, cover?

This section explains why compiling an LLM improves production inference, how common optimizations work, and how to use TensorRT or ONNX Runtime to prepare a model for deployment. Explain how compiling lowers a model from a development framework through an intermediate representation to code optimized for deployment hardware, and why this can improve inference speed and cost. Describe how kernel tuning selects and tests optimized routines for a model’s operations according to the target hardware and input characteristics. Explain how tensor fusion combines operations into a composite kernel to reduce launches and data-handling overhead, and distinguish it from distributing work across GPUs.

What does section 6.1.2, LLM storage strategies, cover?

This section compares ways to store and access an LLM for serving, focusing on startup speed and tradeoffs in reliability, complexity, security, and cost. Explain why storing an LLM in an object store can slow service startup, despite its storage and reliability benefits. Describe how the choice of storage strategy depends on the model, infrastructure, system requirements, and traffic. Describe how fusing mounts a bucket as a filesystem so the service can load a model without a separate download step, and explain why it may still be slow. Explain what it means to bake a model into a Docker image, why this generally fails to improve startup and introduces drawbacks, and when its deployment simplicity may be useful.

What does section 6.1.3, Adaptive request batching, cover?

This section explains how adaptive request batching groups LLM inference requests to improve GPU utilization and throughput, and when its latency cost is worthwhile. Trace how adaptive batching pools incoming requests until a maximum batch size is reached or a timer expires, then runs one inference batch and returns each result to its client. Explain why batching can make GPU computation more efficient. Describe how an inference service can enable dynamic batching and how its maximum batch size and batch delay control when a batch runs. Note that practical defaults often need no further tuning. Justify when adaptive batching is worth using: it generally improves resource use and throughput but adds latency as requests wait. Explain why larger size or delay limits may suit a service with latency budget, while latency-critical or sparse-traffic services may forgo batching.

What does section 6.1.4, Flow control, cover?

This section explains how rate limiters and access keys control traffic to an LLM API, compares four rate-limiting approaches, and discusses how to apply them without unnecessarily disrupting users. Explain how limiting requests over time protects an expensive LLM API from abusive or automated traffic, prevents one client from exhausting resources, and dampens bursts while the service scales or updates. Distinguish fixed window, sliding window log, token bucket, and leaky bucket by how each handles requests, including the fixed-window boundary burst, moving lookback, replenishing tokens, and queue overflow. Explain that access keys authenticate callers, restrict unauthorized or spam use, support paid services, and remain useful for internal APIs because access can be revoked to stop a rogue process and control costs.

What does section 6.1.5, Streaming responses, cover?

This section explains why an LLM service should stream generated text and identifies the components used in a simple streaming implementation. Explain that streaming sends generated text as it becomes available rather than waiting for the complete response, making a slow LLM feel more interactive and reducing the chance that users leave while waiting. Explain that response lengths can vary from a single token to pages, making completion latency unpredictable; streaming shifts attention to tokens per second (TPS), ideally faster than the user's reading speed. Describe how a streaming response signals to users that they are interacting with a bot rather than a human who types and edits replies. Identify asyncio for asynchronous calls, FastAPI's StreamingResponse for sending chunks, and Hugging Face Transformers' TextIteratorStreamer for producing the model-output stream in the example service.

What does section 6.1.6, Feature store, cover?

The section demonstrates how to set up a Feast feature store for a Q&A service and retrieve either current or point-in-time features. Explain how a feature store provides a centralized source of truth for feature ownership, definitions, and access, and why a Q&A service may need a question paired with a timestamp to retrieve an answer that changes over time. Describe the demonstrated workflow: install and initialize Feast, place the Q&A dataset in the project as a Parquet file, define and register a feature view, then materialize it before querying. Explain the view's entities, schema, source, and optional tags and TTL. Distinguish the example's inference-time query for the most up-to-date, low-latency features from point-in-time retrieval with get_historical_features. Describe how the example supplies entity rows as a list of IDs, while an SQL query is also possible.

What does section 6.1.7, Retrieval-augmented generation, cover?

This section explains how RAG retrieves relevant document chunks to ground LLM responses and demonstrates preparing a Pinecone index for that retrieval. Trace how RAG turns document chunks into embeddings stored in a vector database, embeds an incoming question to find similar chunks, and adds the retrieved content to the LLM prompt as context. Describe how the example uses Pinecone to avoid setting up a vector database and uses WikiDataIngestion to load Wikipedia pages, split their text, embed the chunks, and upload them in batches before querying. Explain why the example splits text into 400-token chunks rather than by words or characters, and how retrieving the top three results lets it budget 1,200 tokens for context while leaving room for the user's prompt.

What does section 6.1.8, LLM service libraries, cover?

This section introduces LLM-specific serving libraries, explains what they simplify and where their support varies, and uses vLLM to illustrate starting a local service. Explain that LLM-specific libraries such as vLLM, OpenLLM, and TGI simplify deployment, but differ in supported features and model architectures. Describe streaming, batching, and GPU parallelization as capabilities to expect, while checking other requirements library by library. Describe the section's vLLM example: install the package with pip, start a service for the model trained in chapter 5 with one command, and send requests to its /generate endpoint. Distinguish getting that local service running from setting up infrastructure for production use.

What does section 6.2, Setting up infrastructure, cover?

This section introduces Kubernetes as infrastructure for scalable LLM services and explains the pods, control plane, and custom resources needed to follow the deployment discussion. Explain why Kubernetes is useful for an LLM service: it orchestrates containerized applications across a cluster and helps automate deployment, scaling, and management as demand grows. Describe the simplified Kubernetes layout: containers are grouped into pods, pods run on nodes, and the control plane schedules and manages pod deployment, scaling, and updates. Explain that custom resource definitions let users specify new kinds of Kubernetes resources and then create custom objects like built-in pods or services, without changing Kubernetes source code.

What does section 6.2.1, Provisioning clusters, cover?

This section explains how to provision a cluster for an LLM service, enable GPU-aware node auto-provisioning, and distinguish the resource controls that affect capacity and cost. Describe a cluster as worker machines or nodes that host applications, and explain that creation commands and subsequent setup differ across GCP, AWS, and Azure. State that detailed networking, security, and access-control configuration is outside this section's scope. Explain how node auto-provisioning (NAP), or cluster autoscaling, adds nodes as deployments demand them so unused nodes need not be paid for. Describe why its defaults may need changing for LLM services: the allowed instance pool often excludes GPU accelerators, so the configuration must permit the desired GPU types. Distinguish internal NAP resource limits, which cap scaling and specify expected GPU types, from cloud-provider quotas, which cap resource consumption to help prevent unexpected charges. Explain that raising a quota may require a provider request and that quota headroom does not guarantee regional resource availability.

What does section 6.2.2, Autoscaling, cover?

This section explains why LLM services need autoscaling, how to scale Kubernetes pods using GPU or other custom metrics, and how to tune scaling to balance reliability and cost. Explain why an LLM service should add replicas as demand rises and remove them as demand falls, and why configuring that behavior requires care. Trace how GPU metrics become available for Kubernetes autoscaling: DCGM exports them, Prometheus collects them, and KEDA exposes them to an HPA through a metrics API. Distinguish this setup from the HPA’s native CPU and memory metrics. Explain how to choose a scaling target that reflects the service’s actual bottleneck, and how averaging GPU utilization across devices or time can prevent unstable scaling. Describe when an upstream queue or workflow signal may be more useful than GPU utilization.

What should I know before reading chapter 6?

Transfer learning (finetuning) (chapter 5): Finetuning a model before it is deployed.; Latency (chapter 3): Latency as an operational challenge for LLMs.; Long download times (chapter 3): The operational problem of long LLM download times.; Deployment service (chapter 3): A deployment service serves model inference requests.; Controlling costs (chapter 3): The need to control the cost of operating an LLM.; Feature stores (chapter 3): The earlier introduction to feature stores; Vector databases (chapter 3): Introduces vector databases.; Token limits create bottlenecks (chapter 3): Introduces token limits as an operational bottleneck.

Chapter 6: Large language model services: A practical guide

What you will learn in chapter 6 of LLMs in Production

This introduction defines production as putting an LLM to work for real users and previews the chapter’s path from serving and cloud deployment to production challenges and edge devices.

After this chapter you will be able to

Meaning and value of production. Explain that production means integrating a model into a live environment where it performs tasks or serves end users, making it available for real-world use rather than leaving it as an unused project.
API planning before deployment. Explain why the team should plan an LLM API architecture before deploying a trained model: to define functionality, anticipate integration challenges, arrange resources, and set development priorities.
Service abstraction and latency tradeoff. Describe how a basic LLM service lets users interact with a model on demand and can serve batch or streaming processes without embedding LLM logic in each pipeline. Explain that service communication adds latency, but the abstraction is often worth it because LLMs are already relatively slow.
Model compilation pipeline. Explain how compiling lowers a model from a development framework through an intermediate representation to code optimized for deployment hardware, and why this can improve inference speed and cost.
TensorRT compilation workflow. Trace the section’s TensorRT example from loading a model and example input through TorchScript tracing, TensorRT compilation, and reloading for inference; explain why target hardware and model support matter.
ONNX deployment alternative. Distinguish the ONNX model format from ONNX Runtime, and describe how Hugging Face Optimum exports and optimizes a model as an alternative to TensorRT.
Model storage and boot time. Explain why storing an LLM in an object store can slow service startup, despite its storage and reliability benefits. Describe how the choice of storage strategy depends on the model, infrastructure, system requirements, and traffic.
Direct mounted model volume. Describe serving a model from a mounted file-based volume rather than an object store, including its startup benefit and the coordination, reliability, and cost tradeoffs.
Cached intermediary model volume. Trace the hybrid approach in which the first deployment downloads a model into a mounted volume and later instances reuse it. Explain both its benefit for autoscaling and its limitation after a total system failure.
Adaptive batching request flow. Trace how adaptive batching pools incoming requests until a maximum batch size is reached or a timer expires, then runs one inference batch and returns each result to its client. Explain why batching can make GPU computation more efficient.
Throughput and latency tradeoff. Justify when adaptive batching is worth using: it generally improves resource use and throughput but adds latency as requests wait. Explain why larger size or delay limits may suit a service with latency budget, while latency-critical or sparse-traffic services may forgo batching.
Rate limiting for LLM services. Explain how limiting requests over time protects an expensive LLM API from abusive or automated traffic, prevents one client from exhausting resources, and dampens bursts while the service scales or updates.
Four rate limiter types. Distinguish fixed window, sliding window log, token bucket, and leaky bucket by how each handles requests, including the fixed-window boundary burst, moving lookback, replenishing tokens, and queue overflow.
Access keys for API control. Explain that access keys authenticate callers, restrict unauthorized or spam use, support paid services, and remain useful for internal APIs because access can be revoked to stop a rogue process and control costs.
Streaming and user experience. Explain that streaming sends generated text as it becomes available rather than waiting for the complete response, making a slow LLM feel more interactive and reducing the chance that users leave while waiting.
Variable latency and reading speed. Explain that response lengths can vary from a single token to pages, making completion latency unpredictable; streaming shifts attention to tokens per second (TPS), ideally faster than the user's reading speed.
Feature governance and time. Explain how a feature store provides a centralized source of truth for feature ownership, definitions, and access, and why a Q&A service may need a question paired with a timestamp to retrieve an answer that changes over time.
Feast feature view workflow. Describe the demonstrated workflow: install and initialize Feast, place the Q&A dataset in the project as a Parquet file, define and register a feature view, then materialize it before querying. Explain the view's entities, schema, source, and optional tags and TTL.
Current versus historical retrieval. Distinguish the example's inference-time query for the most up-to-date, low-latency features from point-in-time retrieval with get_historical_features. Describe how the example supplies entity rows as a list of IDs, while an SQL query is also possible.
RAG retrieval and generation flow. Trace how RAG turns document chunks into embeddings stored in a vector database, embeds an incoming question to find similar chunks, and adds the retrieved content to the LLM prompt as context.
Retrieval versus grounded answers. Distinguish the vector database's job of finding articles by embedding proximity from the LLM's job of answering with those articles as context; explain how relevant context and source URLs can reduce made-up answers or references.
LLM serving library tradeoffs. Explain that LLM-specific libraries such as vLLM, OpenLLM, and TGI simplify deployment, but differ in supported features and model architectures. Describe streaming, batching, and GPU parallelization as capabilities to expect, while checking other requirements library by library.
Kubernetes infrastructure purpose. Explain why Kubernetes is useful for an LLM service: it orchestrates containerized applications across a cluster and helps automate deployment, scaling, and management as demand grows.
Pods and control plane. Describe the simplified Kubernetes layout: containers are grouped into pods, pods run on nodes, and the control plane schedules and manages pod deployment, scaling, and updates.
GPU-aware node auto-provisioning. Explain how node auto-provisioning (NAP), or cluster autoscaling, adds nodes as deployments demand them so unused nodes need not be paid for. Describe why its defaults may need changing for LLM services: the allowed instance pool often excludes GPU accelerators, so the configuration must permit the desired GPU types.
Limits and cloud quotas. Distinguish internal NAP resource limits, which cap scaling and specify expected GPU types, from cloud-provider quotas, which cap resource consumption to help prevent unexpected charges. Explain that raising a quota may require a provider request and that quota headroom does not guarantee regional resource availability.
Custom metric scaling pipeline. Trace how GPU metrics become available for Kubernetes autoscaling: DCGM exports them, Prometheus collects them, and KEDA exposes them to an HPA through a metrics API. Distinguish this setup from the HPA’s native CPU and memory metrics.
Scaling metric selection. Explain how to choose a scaling target that reflects the service’s actual bottleneck, and how averaging GPU utilization across devices or time can prevent unstable scaling. Describe when an upstream queue or workflow signal may be more useful than GPU utilization.
Utilization threshold tuning. Describe how the target threshold affects replica counts and why it should reflect traffic patterns and the time needed to bring capacity online. Explain why thresholds that are too low or too high cause problems, including the effect of a distributed GPU pipeline bubble.
Replica count boundaries. Explain how minimum and maximum pod replicas set the service’s baseline and ceiling, and why each should be chosen against observed traffic without wasting capacity or masking problems.
LLM scaling policy timing. Explain why an LLM’s GPU initialization and model loading require more patient downscaling policies, including a stabilization window and limits on how fast pods are removed.
Rolling update purpose and process. Explain that a rolling update replaces old pod instances with new ones gradually so the application can remain accessible during deployment, while making faulty releases easier to catch and roll back.
Rolling update capacity settings. Describe what Kubernetes’s maxSurge and maxUnavailable control, including that each can be an instance count or a percentage. Interpret the example’s five replicas, maxSurge of 1, and maxUnavailable of 3.
GPU constrained rollout tradeoff. Justify why the example keeps maxSurge low and maxUnavailable high: a large LLM needs eight GPUs per replica, so removing old replicas can free scarce GPU resources for new ones. Explain that this saves reserved capacity at the cost of temporarily reduced availability, whereas provisioning extra GPUs allows a smoother add-first rollout.
Separate encoder and LLM deployment. Explain how a sequential inference graph can deploy an encoder and an LLM as separate nodes rather than one packaged service, and describe the benefits of scaling, updating, and calling them separately.
Independent API and model updates. Explain why separating the application API from the LLM is presented as the biggest benefit of an inference graph.
Seldon graph orchestration. Describe how the Seldon example connects an encoder, LLM, classifier, and combining API, and how its Kubernetes configuration defines the graph and lets an orchestrator route requests among nodes.
Silent model quality decline. Explain why an LLM service needs ML monitoring in addition to ordinary system monitoring: data drift and performance decay can lower output quality while the service continues to run.
Sidecar monitoring off the inference path. Explain why text evaluation or data-quality checks can add latency when placed in the direct inference path, and describe how a Kubernetes sidecar can process logging alongside the main application instead.
Ongoing production challenges. Explain that getting an LLM service running is only the start: deploying and maintaining it brings recurring challenges, including model updates, planning for large loads, poor latency, and acquiring resources. State that the following sections offer tips for handling common problems.
Diagnosis before model updates. Explain why a monitored decline in LLM performance should trigger investigation rather than automatic retraining on the latest data. Describe how to examine failing edge cases, remaining strengths, and changes in incoming prompts before choosing a fix.
Targeted fixes and finetuning effects. Distinguish cases where updating RAG or adjusting prompts may fix a problem from cases that call for finetuning with carefully chosen data. Describe how a model update can affect downstream systems and, in knowledge distillation, require a decision about updating the student or teacher.
Proactive update schedule. Describe a periodic LLM update practice in which the team monitors poor-performing cases and gathers relevant examples between planned updates, rather than relying only on reactive fixes.
Load testing workflow. Describe why an LLM service should be load-tested before production or an expected traffic peak, and outline the staging, request-ramping, measurement, and analysis steps.
Capacity and streaming metrics. Distinguish latency and throughput from time to first token and tokens per second, and explain why a streaming LLM service needs both kinds of measures. Describe the role of CPU, GPU, and memory metrics.
Bottleneck and replica planning. Explain how rising response times and falling QPS under increasing users reveal a bottleneck, and how throughput results help estimate the replicas needed for peak load. Distinguish adding replicas for throughput from fixing a latency shortfall.
Network transmission and gRPC. Explain why network transmission can limit LLM latency and throughput, especially for long prompts, and how gRPC uses Protocol Buffers to send serialized binary payloads instead of REST-style JSON.
Profiling and repeat-request caching. Describe using load tests and profiling to locate the slowest part of the service before optimizing code, and using a cache to avoid inference when requests repeat and inputs can be normalized.
GPU scarcity and resource planning. Explain why the section recommends managing GPU resources deliberately: demand for high-performance computing remains high, and LLMs and generative AI add to that demand.
SkyPilot job placement. Describe how SkyPilot runs a defined task through its CLI, searching configured cloud providers, clusters, regions, and zones for an instance that meets the job’s resource requirements. Explain that it can simplify provisioning a GPU-backed Jupyter notebook and set up port forwarding for browser access.
Output token cost. Explain that generating more output tokens costs more, whether the charge comes directly from an API or indirectly through longer inference and additional compute on a self-hosted service. Describe concise prompting as one way to reduce that cost.
Retrieval before generation. Describe when semantic search or stored answers can replace an LLM-generated response: simple fact lookups may return vector-store results directly, and frequently repeated questions may use saved LLM results. Explain why this can be cheaper than generating every answer.
Task and architecture fit. Explain why a smaller model may be cheaper when it performs a particular task well enough, rather than using the largest model for every request. Describe how support for quantization and compilation can matter more to production cost than benchmark results or parameter count alone.
Load tested cost comparison. Describe the section's way to find a cost-optimal deployment: make a matrix of candidate models, services, machines, providers, and prompts; run each combination under load; then combine throughput and running cost to compare dollars per token. Distinguish the fastest option from the most cost-effective one.
Agent execution and containment. Explain why giving an LLM agent tools such as email or web browsing creates an execution risk, and how sandboxing limits what an attacker can make it do.
Layered prompt injection defenses. Describe prompt injection as an attempt to induce unauthorized actions or disclosure, and explain the section’s layered but imperfect defenses: filtering inputs or outputs, validating and sanitizing incoming text, monitoring, and updating or training models to refuse attacks.
Sensitive data exposure assumption. Justify treating any sensitive data used to train an LLM or made accessible to it, including through a VectorDB, as potentially extractable; describe securing the model and its access with measures such as internal use and least privilege.
Edge inference constraints. Explain why edge LLM work focuses on development and inference rather than training, and why limited memory and processing speed shape deployment choices.
Raspberry Pi accelerator path. Describe how a USB TPU such as Coral can improve Raspberry Pi inference, and explain the TensorFlow and TFLite conversion needed to use a TPU.
Quantized CPU model formats. Explain how llama.cpp and its GGUF format make converted Hugging Face models practical for CPU inference on edge devices, and contrast its accessibility with GPTQ's performance focus.

liveBook manning.com

Read chapter 6 on liveBook Get the book at manning.com

What it builds on

Transfer learning (finetuning) (chapter 5) · Finetuning a model before it is deployed.
Latency (chapter 3) · Latency as an operational challenge for LLMs.
Long download times (chapter 3) · The operational problem of long LLM download times.
Deployment service (chapter 3) · A deployment service serves model inference requests.
Controlling costs (chapter 3) · The need to control the cost of operating an LLM.
Feature stores (chapter 3) · The earlier introduction to feature stores
Vector databases (chapter 3) · Introduces vector databases.
Token limits create bottlenecks (chapter 3) · Introduces token limits as an operational bottleneck.

Section by section

Introduction

This introduction defines production as putting an LLM to work for real users and previews the chapter’s path from serving and cloud deployment to production challenges and edge devices.

Meaning and value of production: Explain that production means integrating a model into a live environment where it performs tasks or serves end users, making it available for real-world use rather than leaving it as an unused project.
Why it matters: Deploying a model matters because people can actually use it, even if the model is imperfect.
Chapter deployment path: Describe the chapter’s planned progression: package an LLM as a service or API for on-demand requests, prepare a cloud Kubernetes cluster for deployment, address production challenges, and consider deployment on edge devices.
Why it matters: This roadmap connects serving a model to the infrastructure and operational work needed to make it available.

Key terms: production, live or operational environment, end users, service or API, on-demand requests, Kubernetes cluster, production challenges, edge devices

6.1 Creating an LLM service with RAG and more

Introduces why an LLM API should be planned before deployment, what a basic service enables, and the service features the section will examine.

API planning before deployment: Explain why the team should plan an LLM API architecture before deploying a trained model: to define functionality, anticipate integration challenges, arrange resources, and set development priorities.
Why it matters: Planning makes deployment work easier to coordinate before implementation begins.
Service abstraction and latency tradeoff: Describe how a basic LLM service lets users interact with a model on demand and can serve batch or streaming processes without embedding LLM logic in each pipeline. Explain that service communication adds latency, but the abstraction is often worth it because LLMs are already relatively slow.
Why it matters: This tradeoff explains why a separate model service can be useful despite adding a network call.
Hidden API service features: Describe the basic service diagram as a simplified view that hides work in the API and preprocessing layers, including batching, rate limiting, streaming, and retrieval-augmented generation. State that the section first turns to preparing the model for online inference before working through these service features.
Why it matters: A tidy architecture diagram does not show all the capabilities needed to make the service useful.

Key terms: LLM API, API architecture, integration challenges, LLM service, batch, stream, communication latency, API, batching, rate limiters

6.1.1 Model compilation

This section explains why compiling an LLM improves production inference, how common optimizations work, and how to use TensorRT or ONNX Runtime to prepare a model for deployment.

Model compilation pipeline: Explain how compiling lowers a model from a development framework through an intermediate representation to code optimized for deployment hardware, and why this can improve inference speed and cost.
Why it matters: The compilation pipeline connects model development to efficient production inference.
Hardware-specific kernel tuning: Describe how kernel tuning selects and tests optimized routines for a model’s operations according to the target hardware and input characteristics.
Why it matters: Kernel selection is one way compilation adapts inference to a particular device and workload.
Tensor fusion efficiency: Explain how tensor fusion combines operations into a composite kernel to reduce launches and data-handling overhead, and distinguish it from distributing work across GPUs.
Why it matters: Fusing operations addresses overhead that can remain even when individual GPU computations are fast.
Computational graph restructuring: Explain how graph optimization makes the same computation more efficient by restructuring a graph, including horizontal optimization of layers sharing inputs or constant folding.
Why it matters: Graph restructuring can reduce runtime work without changing what the model computes.
TensorRT compilation workflow: Trace the section’s TensorRT example from loading a model and example input through TorchScript tracing, TensorRT compilation, and reloading for inference; explain why target hardware and model support matter.
Why it matters: The example turns compilation concepts into a deployment workflow while showing its compatibility limits.
ONNX deployment alternative: Distinguish the ONNX model format from ONNX Runtime, and describe how Hugging Face Optimum exports and optimizes a model as an alternative to TensorRT.
Why it matters: ONNX provides a more portable route when an NVIDIA-specific compiler is unsuitable.

Key terms: compiling, lowering, intermediate representation, IR, TorchScript, MLIR, kernel, kernel tuning, GPU, batch size

6.1.2 LLM storage strategies

This section compares ways to store and access an LLM for serving, focusing on startup speed and tradeoffs in reliability, complexity, security, and cost.

Model storage and boot time: Explain why storing an LLM in an object store can slow service startup, despite its storage and reliability benefits. Describe how the choice of storage strategy depends on the model, infrastructure, system requirements, and traffic.
Why it matters: A service cannot use its model until it has retrieved and loaded it, so storage choices affect how quickly new instances become available.
Fused object storage: Describe how fusing mounts a bucket as a filesystem so the service can load a model without a separate download step, and explain why it may still be slow.
Why it matters: Fusing simplifies model-loading code without eliminating the underlying cost of retrieving data from object storage.
Model baked into images: Explain what it means to bake a model into a Docker image, why this generally fails to improve startup and introduces drawbacks, and when its deployment simplicity may be useful.
Why it matters: A self-contained image is convenient, but that convenience should not be mistaken for a faster or safer way to start a service.
Direct mounted model volume: Describe serving a model from a mounted file-based volume rather than an object store, including its startup benefit and the coordination, reliability, and cost tradeoffs.
Why it matters: A nearby mounted volume can make new instances ready quickly, but the team must take on storage management that an object store normally handles.
Cached intermediary model volume: Trace the hybrid approach in which the first deployment downloads a model into a mounted volume and later instances reuse it. Explain both its benefit for autoscaling and its limitation after a total system failure.
Why it matters: The hybrid approach can speed up additional instances without requiring every instance to download the model, provided the shared copy remains available.

Key terms: object storage, artifact registry, bucket, boot times, fusing, FUSE, baking, Docker image, container, mounted volume

6.1.3 Adaptive request batching

This section explains how adaptive request batching groups LLM inference requests to improve GPU utilization and throughput, and when its latency cost is worthwhile.

Adaptive batching request flow: Trace how adaptive batching pools incoming requests until a maximum batch size is reached or a timer expires, then runs one inference batch and returns each result to its client. Explain why batching can make GPU computation more efficient.
Why it matters: Following the request flow makes the throughput gain and added wait time easier to understand.
Batch size and delay controls: Describe how an inference service can enable dynamic batching and how its maximum batch size and batch delay control when a batch runs. Note that practical defaults often need no further tuning.
Why it matters: Existing inference services can provide batching without requiring you to build the queue yourself.
Throughput and latency tradeoff: Justify when adaptive batching is worth using: it generally improves resource use and throughput but adds latency as requests wait. Explain why larger size or delay limits may suit a service with latency budget, while latency-critical or sparse-traffic services may forgo batching.
Why it matters: The right batching choice depends on the product’s tolerance for waiting as well as its traffic.

Key terms: adaptive request batching, dynamic batching, maximum batch size, batch delay, GPU, BentoML, @bentoml.Runnable.method(batchable=True), Triton Inference Server, dynamic_batching {}, throughput

6.1.4 Flow control

This section explains how rate limiters and access keys control traffic to an LLM API, compares four rate-limiting approaches, and discusses how to apply them without unnecessarily disrupting users.

Rate limiting for LLM services: Explain how limiting requests over time protects an expensive LLM API from abusive or automated traffic, prevents one client from exhausting resources, and dampens bursts while the service scales or updates.
Why it matters: Traffic control helps keep an LLM service available and its resources fairly shared.
Four rate limiter types: Distinguish fixed window, sliding window log, token bucket, and leaky bucket by how each handles requests, including the fixed-window boundary burst, moving lookback, replenishing tokens, and queue overflow.
Why it matters: Different limiters handle bursts differently, so the choice depends on the service's needs.
Access keys for API control: Explain that access keys authenticate callers, restrict unauthorized or spam use, support paid services, and remain useful for internal APIs because access can be revoked to stop a rogue process and control costs.
Why it matters: Knowing who can call an API lets a service block misuse and revoke costly access.
Limiter placement and tuning: Describe where rate limits can be applied and how to set them loosely enough to preserve user experience while still stopping harmful traffic, rather than using aggressive limits in place of autoscaling. State that the section's FastAPI example uses built-in security for access keys and SlowApi decorators for limits, but a real service still needs key creation and destruction.
Why it matters: A protective limit is useful only if it does not needlessly block legitimate users.

Key terms: rate limiters, DDoS, quotas, bursty workloads, fixed window, sliding window log, token bucket, leaky bucket, access keys, authentication

6.1.5 Streaming responses

This section explains why an LLM service should stream generated text and identifies the components used in a simple streaming implementation.

Streaming and user experience: Explain that streaming sends generated text as it becomes available rather than waiting for the complete response, making a slow LLM feel more interactive and reducing the chance that users leave while waiting.
Why it matters: Showing progress keeps users engaged even when the full response takes time.
Variable latency and reading speed: Explain that response lengths can vary from a single token to pages, making completion latency unpredictable; streaming shifts attention to tokens per second (TPS), ideally faster than the user's reading speed.
Why it matters: A steady stream that outpaces reading can preserve the user's flow despite unpredictable completion times.
Bot signaling and service components: Describe how a streaming response signals to users that they are interacting with a bot rather than a human who types and edits replies. Identify asyncio for asynchronous calls, FastAPI's StreamingResponse for sending chunks, and Hugging Face Transformers' TextIteratorStreamer for producing the model-output stream in the example service.
Why it matters: Streaming both sets user expectations and requires the service to deliver model output in chunks.

Key terms: streaming, generated text, latency, tokens per second, TPS, reading speed, asyncio, FastAPI, StreamingResponse, TextIteratorStreamer

6.1.6 Feature store

The section demonstrates how to set up a Feast feature store for a Q&A service and retrieve either current or point-in-time features.

Feature governance and time: Explain how a feature store provides a centralized source of truth for feature ownership, definitions, and access, and why a Q&A service may need a question paired with a timestamp to retrieve an answer that changes over time.
Why it matters: A governed, time-aware source helps a service return the answer appropriate to the question and date.
Feast feature view workflow: Describe the demonstrated workflow: install and initialize Feast, place the Q&A dataset in the project as a Parquet file, define and register a feature view, then materialize it before querying. Explain the view's entities, schema, source, and optional tags and TTL.
Why it matters: Defining a view is not enough; registering and materializing it makes its features available for retrieval.
Current versus historical retrieval: Distinguish the example's inference-time query for the most up-to-date, low-latency features from point-in-time retrieval with get_historical_features. Describe how the example supplies entity rows as a list of IDs, while an SQL query is also possible.
Why it matters: Choosing the retrieval method determines whether the service receives current features or features from a requested time.

Key terms: feature store, data governance, point-in-time joins, Feast, feature view, entities, schema, source, tags, inference time

6.1.7 Retrieval-augmented generation

This section explains how RAG retrieves relevant document chunks to ground LLM responses and demonstrates preparing a Pinecone index for that retrieval.

RAG retrieval and generation flow: Trace how RAG turns document chunks into embeddings stored in a vector database, embeds an incoming question to find similar chunks, and adds the retrieved content to the LLM prompt as context.
Why it matters: The retrieval-to-prompt flow is what makes RAG useful for grounding an answer in relevant material.
Pinecone document ingestion example: Describe how the example uses Pinecone to avoid setting up a vector database and uses WikiDataIngestion to load Wikipedia pages, split their text, embed the chunks, and upload them in batches before querying.
Why it matters: Ingestion makes the document collection searchable when a question arrives.
Token based chunk budgeting: Explain why the example splits text into 400-token chunks rather than by words or characters, and how retrieving the top three results lets it budget 1,200 tokens for context while leaving room for the user's prompt.
Why it matters: Predictable context size helps keep retrieved material within the model's token limits.
Retrieval versus grounded answers: Distinguish the vector database's job of finding articles by embedding proximity from the LLM's job of answering with those articles as context; explain how relevant context and source URLs can reduce made-up answers or references.
Why it matters: RAG improves grounding, but search results alone are not an answer and do not guarantee correctness.

Key terms: retrieval-augmented generation (RAG), embeddings, vector database, similarity search, semantic search, nearest neighbors, Pinecone, Pinecone index, WikiDataIngestion, Wikipedia dataset

6.1.8 LLM service libraries

This section introduces LLM-specific serving libraries, explains what they simplify and where their support varies, and uses vLLM to illustrate starting a local service.

LLM serving library tradeoffs: Explain that LLM-specific libraries such as vLLM, OpenLLM, and TGI simplify deployment, but differ in supported features and model architectures. Describe streaming, batching, and GPU parallelization as capabilities to expect, while checking other requirements library by library.
Why it matters: A library can get a service running quickly without necessarily meeting every production requirement.
Local vLLM service example: Describe the section's vLLM example: install the package with pip, start a service for the model trained in chapter 5 with one command, and send requests to its /generate endpoint. Distinguish getting that local service running from setting up infrastructure for production use.
Why it matters: A quick local launch is useful, but it does not replace production infrastructure.

Key terms: vLLM, OpenLLM, Text-Generation-Inference (TGI), streaming, batching, GPU parallelization, pip install vllm, /generate endpoint, production infrastructure

6.2 Setting up infrastructure

This section introduces Kubernetes as infrastructure for scalable LLM services and explains the pods, control plane, and custom resources needed to follow the deployment discussion.

Kubernetes infrastructure purpose: Explain why Kubernetes is useful for an LLM service: it orchestrates containerized applications across a cluster and helps automate deployment, scaling, and management as demand grows.
Why it matters: This explains why the chapter uses Kubernetes to run and grow its services.
Pods and control plane: Describe the simplified Kubernetes layout: containers are grouped into pods, pods run on nodes, and the control plane schedules and manages pod deployment, scaling, and updates.
Why it matters: These relationships make the later cluster and rollout discussions easier to follow.
Custom resource definitions: Explain that custom resource definitions let users specify new kinds of Kubernetes resources and then create custom objects like built-in pods or services, without changing Kubernetes source code.
Why it matters: CRDs explain how Kubernetes can be extended for functionality the chapter needs.

Key terms: Kubernetes, k8s, container orchestration, automatic scaling, load balancing, service discovery, containers, pods, nodes, control plane

6.2.1 Provisioning clusters

This section explains how to provision a cluster for an LLM service, enable GPU-aware node auto-provisioning, and distinguish the resource controls that affect capacity and cost.

Cluster setup across providers: Describe a cluster as worker machines or nodes that host applications, and explain that creation commands and subsequent setup differ across GCP, AWS, and Azure. State that detailed networking, security, and access-control configuration is outside this section's scope.
Why it matters: The provider determines the exact setup steps, even when the deployment goal is the same.
GPU-aware node auto-provisioning: Explain how node auto-provisioning (NAP), or cluster autoscaling, adds nodes as deployments demand them so unused nodes need not be paid for. Describe why its defaults may need changing for LLM services: the allowed instance pool often excludes GPU accelerators, so the configuration must permit the desired GPU types.
Why it matters: A cluster that can scale ordinary nodes may still be unable to obtain the GPU nodes an LLM service needs.
Limits and cloud quotas: Distinguish internal NAP resource limits, which cap scaling and specify expected GPU types, from cloud-provider quotas, which cap resource consumption to help prevent unexpected charges. Explain that raising a quota may require a provider request and that quota headroom does not guarantee regional resource availability.
Why it matters: A GPU request can fail even when the cluster is below its configured limit and cloud quota.
Reservations and commitments: Distinguish reservations and commitments from limits and quotas: the former secure a lower bound of resources rather than imposing an upper bound. Explain that reservations guarantee availability and may be charged even when unused, while commitments are typically longer-term agreements with a discount.
Why it matters: Guaranteed capacity can solve availability concerns, but it brings a different cost obligation from a usage cap.

Key terms: cluster, nodes, gcloud, eksctl, az cli, node auto-provisioning, NAP, cluster autoscaling, GPU resourceType, T4

6.2.2 Autoscaling

This section explains why LLM services need autoscaling, how to scale Kubernetes pods using GPU or other custom metrics, and how to tune scaling to balance reliability and cost.

Autoscaling benefits and tradeoffs: Explain why an LLM service should add replicas as demand rises and remove them as demand falls, and why configuring that behavior requires care.
Why it matters: Scaling down saves money, while scaling up helps the service survive peaks and bursts.
Custom metric scaling pipeline: Trace how GPU metrics become available for Kubernetes autoscaling: DCGM exports them, Prometheus collects them, and KEDA exposes them to an HPA through a metrics API. Distinguish this setup from the HPA’s native CPU and memory metrics.
Why it matters: An HPA cannot act on GPU utilization merely because Prometheus has recorded it.
Scaling metric selection: Explain how to choose a scaling target that reflects the service’s actual bottleneck, and how averaging GPU utilization across devices or time can prevent unstable scaling. Describe when an upstream queue or workflow signal may be more useful than GPU utilization.
Why it matters: A poorly chosen or jumpy metric can leave a busy service unscaled or make its replicas flap.
Utilization threshold tuning: Describe how the target threshold affects replica counts and why it should reflect traffic patterns and the time needed to bring capacity online. Explain why thresholds that are too low or too high cause problems, including the effect of a distributed GPU pipeline bubble.
Why it matters: The threshold determines whether new capacity arrives soon enough without causing needless scaling.
Replica count boundaries: Explain how minimum and maximum pod replicas set the service’s baseline and ceiling, and why each should be chosen against observed traffic without wasting capacity or masking problems.
Why it matters: Replica limits protect both availability and the cluster’s resource budget.
LLM scaling policy timing: Explain why an LLM’s GPU initialization and model loading require more patient downscaling policies, including a stabilization window and limits on how fast pods are removed.
Why it matters: Prematurely removing a slow-starting replica can waste its entire startup effort or interrupt work.

Key terms: autoscaling, replicas, bursty workloads, horizontal pod autoscaler, HPA, DCGM, Prometheus, KEDA, custom metrics API, target parameter

6.2.3 Rolling updates

Explains how Kubernetes rolling updates replace an LLM deployment’s pods and how GPU constraints shape the update settings and rollout precautions.

Rolling update purpose and process: Explain that a rolling update replaces old pod instances with new ones gradually so the application can remain accessible during deployment, while making faulty releases easier to catch and roll back.
Why it matters: The replacement process explains why a deployment need not stop serving users while it changes.
Rolling update capacity settings: Describe what Kubernetes’s maxSurge and maxUnavailable control, including that each can be an instance count or a percentage. Interpret the example’s five replicas, maxSurge of 1, and maxUnavailable of 3.
Why it matters: These settings determine how much extra capacity a rollout can use and how much serving capacity it can temporarily lose.
GPU constrained rollout tradeoff: Justify why the example keeps maxSurge low and maxUnavailable high: a large LLM needs eight GPUs per replica, so removing old replicas can free scarce GPU resources for new ones. Explain that this saves reserved capacity at the cost of temporarily reduced availability, whereas provisioning extra GPUs allows a smoother add-first rollout.
Why it matters: LLM GPU requirements can make the safer-looking add-first rollout impractical or expensive.
Staging and spare capacity decisions: Describe why the section recommends testing rolling updates and tuning their settings in staging, while noting that staging may not match production resources. Explain that reserving GPUs solely for smoother model deployments may not pay off when the LLM changes infrequently and most changes concern the API, prompts, or surrounding application.
Why it matters: A rollout policy should be tested before production and weighed against how often it will actually be needed.

Key terms: rolling updates, rolling upgrades, pods, zero downtime, roll back, spec.strategy, maxSurge, maxUnavailable, replicas, GPU resources

6.2.4 Inference graphs

This section explains how inference graphs connect model and API components, what deployment flexibility they offer, and when their added complexity is worthwhile.

Separate encoder and LLM deployment: Explain how a sequential inference graph can deploy an encoder and an LLM as separate nodes rather than one packaged service, and describe the benefits of scaling, updating, and calling them separately.
Why it matters: Separating the models lets a team manage each one according to its workload and reuse the deployed encoder.
Independent API and model updates: Explain why separating the application API from the LLM is presented as the biggest benefit of an inference graph.
Why it matters: Teams can change fast-moving application behavior without repeatedly deploying the LLM.
Seldon graph orchestration: Describe how the Seldon example connects an encoder, LLM, classifier, and combining API, and how its Kubernetes configuration defines the graph and lets an orchestrator route requests among nodes.
Why it matters: The example shows how a multi-component inference flow can be declared and coordinated in a deployment.
Inference graph complexity tradeoff: Justify when an inference graph is worth considering and why the section still recommends keeping deployments simple when possible.
Why it matters: An architecture that simplifies a complex system can still be unnecessary overhead for a simpler one.

Key terms: inference graph, sequential inference graph, encoder, LLM, VectorDB, API, Seldon, SeldonDeployment, Kubernetes CRD, orchestrator

6.2.5 Monitoring

This section explains how to monitor an LLM service for operational failures and silent quality decline without putting costly monitoring work in the inference path.

Operational service and resource metrics: Describe the service metrics and resource metrics to monitor in production, and explain how they help reveal failures that need prompt attention.
Why it matters: These metrics help you spot reliability and performance problems before they undermine the service.
Silent model quality decline: Explain why an LLM service needs ML monitoring in addition to ordinary system monitoring: data drift and performance decay can lower output quality while the service continues to run.
Why it matters: A healthy-looking service can still disappoint users if the language or quality of its outputs changes.
Text monitoring with whylogs and LangKit: Describe how whylogs and LangKit can help monitor incoming prompts and generated text: whylogs captures summary statistics at scale, while LangKit adds LLM-oriented text metrics.
Why it matters: Text-specific signals can expose quality or safety changes that hardware dashboards cannot show.
Sidecar monitoring off the inference path: Explain why text evaluation or data-quality checks can add latency when placed in the direct inference path, and describe how a Kubernetes sidecar can process logging alongside the main application instead.
Why it matters: Moving monitoring work off the request path helps preserve response latency while retaining useful logs.

Key terms: queries per second (QPS), latency, response code counts, CPU utilization, GPU utilization, GPU temperature, silent failures, data drift, performance decay, ML monitoring

6.3 Cost engineering, security, latency, and other production challenges

This introduction shifts from getting an LLM service running to the recurring challenges of deploying and maintaining it in production.

Ongoing production challenges: Explain that getting an LLM service running is only the start: deploying and maintaining it brings recurring challenges, including model updates, planning for large loads, poor latency, and acquiring resources. State that the following sections offer tips for handling common problems.
Why it matters: A working deployment still needs ongoing attention to remain reliable in production.

Key terms: deploying models, maintaining them in production, updating, large loads, poor latency, acquiring resources

6.3.1 Model updates and retraining

The section explains how to respond to declining LLM performance by diagnosing failures, choosing targeted fixes, and planning model updates rather than automatically retraining.

Diagnosis before model updates: Explain why a monitored decline in LLM performance should trigger investigation rather than automatic retraining on the latest data. Describe how to examine failing edge cases, remaining strengths, and changes in incoming prompts before choosing a fix.
Why it matters: A decline in performance does not, by itself, tell you what data or change the model needs.
Targeted fixes and finetuning effects: Distinguish cases where updating RAG or adjusting prompts may fix a problem from cases that call for finetuning with carefully chosen data. Describe how a model update can affect downstream systems and, in knowledge distillation, require a decision about updating the student or teacher.
Why it matters: Choosing the narrowest effective fix avoids unnecessary finetuning and unintended effects elsewhere in the system.
Proactive update schedule: Describe a periodic LLM update practice in which the team monitors poor-performing cases and gathers relevant examples between planned updates, rather than relying only on reactive fixes.
Why it matters: A regular process turns observed failures into useful update data and helps the model improve over time.

Key terms: silent failures, data drift, edge cases, incoming prompts, RAG, finetuning, prompts, knowledge distillation, student model, teacher model

6.3.2 Load testing

This section shows how to load-test an LLM service, measure both capacity and streaming responsiveness, and use Locust results to plan for peak traffic.

Load testing workflow: Describe why an LLM service should be load-tested before production or an expected traffic peak, and outline the staging, request-ramping, measurement, and analysis steps.
Why it matters: Testing early can reveal bottlenecks before users encounter slowdowns or failures.
Capacity and streaming metrics: Distinguish latency and throughput from time to first token and tokens per second, and explain why a streaming LLM service needs both kinds of measures. Describe the role of CPU, GPU, and memory metrics.
Why it matters: A service can appear adequate by completed-request metrics while still feeling slow to someone waiting for streamed output.
Locust test setup: Describe how the section uses Locust to send prompts to a running LLM service, including its web UI and headless option, and explain why streaming TTFT and TPS require custom metrics.
Why it matters: The test setup turns the proposed measurements into a repeatable check against a real service.
Bottleneck and replica planning: Explain how rising response times and falling QPS under increasing users reveal a bottleneck, and how throughput results help estimate the replicas needed for peak load. Distinguish adding replicas for throughput from fixing a latency shortfall.
Why it matters: Recognizing where capacity stops increasing helps plan scaling without mistaking it for a cure for every latency problem.

Key terms: load testing, staging environment, autoscaling, latency, throughput, queries per second, QPS, time to first token, TTFT, Locust

6.3.3 Troubleshooting poor latency

This section shows how to troubleshoot slow LLM services by measuring bottlenecks and reducing network, hardware, and inference latency.

Network transmission and gRPC: Explain why network transmission can limit LLM latency and throughput, especially for long prompts, and how gRPC uses Protocol Buffers to send serialized binary payloads instead of REST-style JSON.
Why it matters: A slow response may be caused by moving data, not by running the model.
gRPC client and API setup: Describe the practical work needed to use gRPC: an inference service may already expose it, but the client must serialize and deserialize messages; building an API without such a service calls for tools such as grpcio and .proto files rather than FastAPI.
Why it matters: The protocol's speed benefit is useful only if the service and client can exchange its message format.
Compilation and hardware placement: Explain how model compilation, deploying near users, and choosing a faster accelerator can improve latency, including the constraint that a nearby data center may lack the desired accelerator and changing accelerator architecture requires recompilation.
Why it matters: A deployment change can reduce one source of delay while creating a hardware or compilation constraint.
Profiling and repeat-request caching: Describe using load tests and profiling to locate the slowest part of the service before optimizing code, and using a cache to avoid inference when requests repeat and inputs can be normalized.
Why it matters: Measurements direct effort toward the actual bottleneck, while a cache can eliminate unnecessary model runs.
Smaller model speed tradeoff: Explain why a smaller model or an ensemble of smaller models may be worth considering when latency is poor, despite a possible loss of model quality or result accuracy.
Why it matters: A modest quality sacrifice may improve the speed and reliability of the overall service.

Key terms: latency, throughput, gRPC, Protocol Buffers, protobufs, REST, Seldon V2 Inference Protocol, MLServer, grpcio, .proto files

6.3.4 Resource management

The section explains why scarce GPU capacity needs active management and introduces SkyPilot and Run:ai as tools for making it more available to jobs.

GPU scarcity and resource planning: Explain why the section recommends managing GPU resources deliberately: demand for high-performance computing remains high, and LLMs and generative AI add to that demand.
Why it matters: GPU availability can determine whether an LLM job can run when you need it.
SkyPilot job placement: Describe how SkyPilot runs a defined task through its CLI, searching configured cloud providers, clusters, regions, and zones for an instance that meets the job’s resource requirements. Explain that it can simplify provisioning a GPU-backed Jupyter notebook and set up port forwarding for browser access.
Why it matters: This is the section’s concrete example of finding available GPU capacity without manually provisioning each option.
Run:ai accelerator management: Describe Run:ai as another GPU optimization and cluster-management option, citing examples such as fractional GPUs or oversubscription and GPU pooling or job scheduling. Distinguish these advertised capabilities from the section’s more concrete SkyPilot example.
Why it matters: It shows that resource management also includes sharing and scheduling accelerator capacity within clusters.

Key terms: GPU shortage, high-performance computing, resource management, SkyPilot, sky CLI, resource requirements, GPU-backed Jupyter notebook, port forwarding, Run:ai, GPU oversubscription

6.3.5 Cost engineering

This section explains how to reduce LLM serving costs and compare deployment options using cost per token under load.

Output token cost: Explain that generating more output tokens costs more, whether the charge comes directly from an API or indirectly through longer inference and additional compute on a self-hosted service. Describe concise prompting as one way to reduce that cost.
Why it matters: Output length is a cost lever for both purchased and self-hosted inference.
Retrieval before generation: Describe when semantic search or stored answers can replace an LLM-generated response: simple fact lookups may return vector-store results directly, and frequently repeated questions may use saved LLM results. Explain why this can be cheaper than generating every answer.
Why it matters: Not every user question needs a fresh LLM response.
Task and architecture fit: Explain why a smaller model may be cheaper when it performs a particular task well enough, rather than using the largest model for every request. Describe how support for quantization and compilation can matter more to production cost than benchmark results or parameter count alone.
Why it matters: Model selection should account for the task and production efficiency, not just headline capability.
GPU configuration tradeoffs: Explain why GPU selection should consider model memory fit and processing efficiency, not just the hourly price of each device. Describe how splitting a model across cheaper GPUs can introduce inefficiency, while a more expensive GPU may finish work faster and cost less overall.
Why it matters: The cheapest hardware price does not necessarily produce the cheapest inference.
Load tested cost comparison: Describe the section's way to find a cost-optimal deployment: make a matrix of candidate models, services, machines, providers, and prompts; run each combination under load; then combine throughput and running cost to compare dollars per token. Distinguish the fastest option from the most cost-effective one.
Why it matters: Measured cost under realistic load provides a firmer basis for deployment decisions than rules of thumb.

Key terms: output tokens, inference times, compute time, text embeddings, semantic search, vector store, neural information retrieval, RAG, smaller model, parameters

6.3.6 Security

This section explains the main security risks of production LLMs, why prompt-injection defenses are imperfect, and how to limit the damage if an attack succeeds.

Agent execution and containment: Explain why giving an LLM agent tools such as email or web browsing creates an execution risk, and how sandboxing limits what an attacker can make it do.
Why it matters: An injected instruction is more dangerous when the model can act on it through a tool.
Layered prompt injection defenses: Describe prompt injection as an attempt to induce unauthorized actions or disclosure, and explain the section’s layered but imperfect defenses: filtering inputs or outputs, validating and sanitizing incoming text, monitoring, and updating or training models to refuse attacks.
Why it matters: No single filter reliably stops a determined attacker, so production defenses must work in layers.
Multilingual safeguard bypasses: Explain how attacks in unsupported or mixed languages can bypass safeguards built primarily for English, and describe the trade-offs of rejecting unsupported languages versus flagging them for closer monitoring.
Why it matters: Language policies affect both the reach of an attack and whether legitimate users can use the service.
Adversarial probing and poisoning: Distinguish an attacker’s repeated prompt probing and response collection from data poisoning: the former helps predict prompts that elicit a desired output, while the latter plants malicious training data that changes later model behavior.
Why it matters: Attacks can exploit either the model’s observed responses or the data used to train it.
Sensitive data exposure assumption: Justify treating any sensitive data used to train an LLM or made accessible to it, including through a VectorDB, as potentially extractable; describe securing the model and its access with measures such as internal use and least privilege.
Why it matters: Safeguards make extraction harder, but they do not turn a model with access to secrets into a safe place to expose publicly.

Key terms: LLM agent, tooling, plugins, sandboxed, prompt injection, context-aware filtering, keyword search, second LLM, monitoring, sanitization

6.4 Deploying to the edge

This section outlines practical hardware and model-format choices for running LLM inference on memory- and speed-constrained edge devices.

Edge inference constraints: Explain why edge LLM work focuses on development and inference rather than training, and why limited memory and processing speed shape deployment choices.
Why it matters: These constraints explain why edge deployment needs different hardware or model formats.
Raspberry Pi accelerator path: Describe how a USB TPU such as Coral can improve Raspberry Pi inference, and explain the TensorFlow and TFLite conversion needed to use a TPU.
Why it matters: Adding an accelerator helps only if the model and inference code can run on it.
Alternative edge hardware choices: Distinguish a Jetson-like board with a built-in GPU and CUDA from the Raspberry Pi plus TPU path, and identify phones as another possible edge target.
Why it matters: The hardware choice affects both the development workflow and the accelerator available.
Quantized CPU model formats: Explain how llama.cpp and its GGUF format make converted Hugging Face models practical for CPU inference on edge devices, and contrast its accessibility with GPTQ's performance focus.
Why it matters: Model format and inference software can make a limited device useful even without a dedicated accelerator.

Key terms: edge devices, memory, speed, inference, Raspberry Pi, Coral, USB-TPU, Optimum, .tflite, TensorFlow

6.5 Summary

The section recaps practical choices for serving LLMs, managing their infrastructure, and addressing production and edge-deployment challenges.

Where this is used later

(chapter intro) (chapter 8) · Creating an LLM service.
Building an application with RAG (chapter 8) · The LLM service that an application calls.
Our model (chapter 10) · serving an LLM through an API
Streaming on the frontend (chapter 8) · The application presents streamed responses on the frontend.
RAG applied (chapter 8) · Applies RAG in an application.
Using RAG (chapter 10) · Uses RAG in the coding copilot project.
Our VectorDB (chapter 10) · Sets up a vector database for a RAG project.
Prompt engineering basics (chapter 7) · RAG is a technique that can be used alongside prompting.

Concepts explained in this chapter

DCGMData Center GPU Manager is NVIDIA software for managing, monitoring, and collecting telemetry from data-center GPUs.
GPU metricA GPU metric is an operational measurement describing graphics-processor utilization, capacity, or behavior.
KEDAKEDA is an open-source Kubernetes project that scales workloads according to event sources and custom or external metrics.
PrometheusPrometheus is an open-source monitoring system that collects time-series metrics and supports querying and alerting.
scalabilityAutoscaling is the automatic adjustment of computing resources or service instances in response to workload or capacity conditions.
HorizontalPodAutoscalerA horizontal pod autoscaler is a Kubernetes controller that adjusts an application's pod replica count according to observed metrics.
Prometheus AdapterPrometheus Adapter is a Kubernetes metrics adapter that exposes Prometheus metrics through Kubernetes metrics APIs.
ScaledObjectA ScaledObject is a KEDA custom resource that declares scaling behavior and triggers for a Kubernetes workload.
custom metrics APIThe custom metrics API is a Kubernetes API for exposing application-defined metrics associated with Kubernetes resources.
flappingFlapping is repeated oscillation between scaling states, such as repeatedly adding and removing service replicas.
GPU kernelA kernel is a small program or function executed on a parallel processor to perform a computational operation.
ONNXONNX is an open model format and ecosystem for representing neural-network models across different frameworks, tools, and hardware environments.
ONNX RuntimeONNX Runtime is a cross-platform inference engine that optimizes and executes models represented in ONNX format.
TensorRTTensorRT is NVIDIA software for optimizing and executing machine-learning inference on NVIDIA GPUs.
code compilationCompilation is the transformation of high-level program or model code into lower-level code optimized for execution on target hardware.
intermediate representationAn intermediate representation is a machine-readable form of a program or model that enables translation and optimization between source frameworks and target hardware.
kernel tuningKernel tuning is the selection and optimization of processor kernels to minimize execution time for a specific workload and hardware configuration.
tensor fusionTensor fusion is the merging of multiple tensor-operation kernels into a combined operation to reduce launches and improve memory efficiency.
trajectoryModel tracing is the capture of operations invoked during a model execution to construct an intermediate computational representation.
throughputThroughput is the rate at which a system completes requests or produces outputs over a period of time.
rate limitA rate limiter is a mechanism that restricts how many requests a client can make to a service within a specified time interval.
AI agentA software system that uses artificial intelligence to perceive information, make decisions, and perform actions toward goals.
adversarial inputThe study and use of inputs or models designed to manipulate, evade, or expose weaknesses in machine-learning systems.
context-aware filteringContext-aware filtering evaluates input or output in its surrounding semantic context to detect unsafe, unauthorized, or policy-violating content.
poisoning attackData poisoning is an attack that inserts malicious or manipulated examples into training data to alter a model's behavior.
prompt injectionPrompt injection is an attack in which crafted input attempts to manipulate a language model into ignoring intended instructions or revealing unintended information.
securitySecurity is the protection of systems, data, and operations against unauthorized access, misuse, disruption, or compromise.
object storeAn object storage system stores data as independently addressable objects distributed across storage infrastructure.
baking modelBaking a model is the technique of packaging model files inside a container image so newly created containers contain the model.
boot timeBoot time is the elapsed time required for a machine or service instance to start and become ready for operation.
fusingFusing is the technique of mounting remote object storage on a machine so it appears and behaves like a local filesystem.
intermediary mounted volumeAn intermediary mounted volume is a storage strategy in which a model is downloaded once and retained on a volume that later instances can mount.
mounted volumeA mounted volume is a storage resource attached to a running machine or container and exposed through its filesystem.
streamingStreaming is the incremental retrieval and processing of a dataset without loading the entire dataset into memory.
LocustLocust is an open-source load-testing tool for defining and executing distributed user-behavior simulations.
load testingLoad testing is a performance-testing technique that evaluates how a service or system behaves under an expected or increasing workload.
replicaA replica is an additional running instance of a service or workload that provides capacity or redundancy.
time to first tokenTime to first token is the elapsed time between submitting a language-model request and receiving its first generated token.
batch sizeBatch size is the number of input examples or inference requests processed together in one computation batch.
cachingCaching is the technique of storing reusable computation results or data so later requests can be served without repeating the original work.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.