What will I learn in chapter 3 of LLMs in Production?

The introduction explains why larger language models require scaled-up operations and infrastructure, then previews the chapter’s focus on deployment challenges, ways to reduce them, and supporting architecture. After reading it you should be able to: Explain that although bigger, linguistically informed transformer models can be more effective for NLP, their size creates operational and infrastructure challenges regardless of their linguistic efficacy. Explain that the section treats LLMOps as MLOps scaled to handle LLMs, while questioning whether it is distinct enough to need its own term. Describe why a rudimentary MLOps setup may not be able to handle that scale. Describe MLOps as reliably and efficiently deploying and maintaining ML models in production across the whole lifecycle, from data acquisition and training through monitoring and termination. Name representative practices the section says this requires, such as orchestration, versioning, feedback loops, CI/CD, security, resource provisioning, or data governance. Explain that LLMOps overlaps with MLOps but is not merely a fashionable new name: LLMs bring operational problems uncommon in traditional machine learning, largely because the models are so big. Explain why downloading a large LLM must be planned for as part of production work: Bloom is about 330 GB, and downloading it took the author's team about an hour and a half. Explain why practices that work for small models—downloading at run time and relying on quick application restarts—are poor fits for large LLMs. A shutdown may take seconds while redeployment can take much longer. Explain how long deployment times make scaling down before a traffic surge and handling bursty workloads difficult. Describe how slower rolling updates create CI/CD backlogs and make mistakes slower to notice and correct. Explain how larger models tend to increase inference latency, and why streaming a response one word at a time can make a chatbot feel faster without eliminating the wait. Describe why that latency may make LLMs unsuitable for fast autocomplete or large-scale text-processing workflows.

What does section 3.1, Introduction to large language model operations, cover?

Introduces LLMOps as MLOps scaled for large language models, then explains the production lifecycle and practices that MLOps covers. Explain that the section treats LLMOps as MLOps scaled to handle LLMs, while questioning whether it is distinct enough to need its own term. Describe why a rudimentary MLOps setup may not be able to handle that scale. Describe MLOps as reliably and efficiently deploying and maintaining ML models in production across the whole lifecycle, from data acquisition and training through monitoring and termination. Name representative practices the section says this requires, such as orchestration, versioning, feedback loops, CI/CD, security, resource provisioning, or data governance.

What does section 3.2, Operations challenges with large language models, cover?

This introduction explains why LLMOps merits a distinction from MLOps and frames the operational challenges of deploying large models. Explain that LLMOps overlaps with MLOps but is not merely a fashionable new name: LLMs bring operational problems uncommon in traditional machine learning, largely because the models are so big.

What does section 3.2.1, Long download times, cover?

This section shows why the sheer size of LLMs makes downloading model files a production bottleneck, even when parallel downloads help. Explain why downloading a large LLM must be planned for as part of production work: Bloom is about 330 GB, and downloading it took the author's team about an hour and a half. Distinguish an ideal download-time estimate from a realistic one, and describe how splitting Bloom into smaller files permits parallel downloads without eliminating the delay.

What does section 3.2.2, Longer deploy times, cover?

Large LLMs take long enough to deploy that teams must rethink runtime downloads, restarts, scaling, and release workflows. Describe why deploying a large LLM can take much longer than downloading it: loading the model into GPU memory takes time, other deployment steps add time, and teams may have to wait for GPUs to become available. Explain why practices that work for small models—downloading at run time and relying on quick application restarts—are poor fits for large LLMs. A shutdown may take seconds while redeployment can take much longer. Explain how long deployment times make scaling down before a traffic surge and handling bursty workloads difficult. Describe how slower rolling updates create CI/CD backlogs and make mistakes slower to notice and correct.

What does section 3.2.3, Latency, cover?

This section explains why LLM inference latency can be a production bottleneck even when a streamed chatbot response feels fast. Explain how larger models tend to increase inference latency, and why streaming a response one word at a time can make a chatbot feel faster without eliminating the wait. Describe why that latency may make LLMs unsuitable for fast autocomplete or large-scale text-processing workflows. Explain that distributing an LLM across GPUs adds communication overhead, even when distribution is intended to improve latency. Describe how longer completions take longer to produce and how chain-of-thought prompting can trade additional response time for better results on logic or math questions.

What does section 3.2.4, Managing GPUs, cover?

This section explains why LLMs need GPUs and why adding them to production systems creates tooling, provisioning, and latency challenges. Explain why LLM inference and training typically need GPUs rather than relying on CPUs: GPUs accelerate parallel processing and the linear algebra, matrix multiplication, and tensor operations underlying LLMs. Describe the operational changes introduced when an LLM service needs GPUs: architectures and orchestration tools often assume CPU and memory, so GPU support may require rebuilding containers and deploying new metrics systems. Explain how limited GPU supply complicates both on-premise and cloud deployments: on-premise servers may lack accessible GPUs, while cloud GPU types can be exhausted or unavailable in a region, potentially forcing deployment farther from users and increasing latency.

What does section 3.2.5, Peculiarities of text data, cover?

The section explains why the meaning of text is harder to encode numerically than physical measurements, and how that difficulty complicates LLM operations. Distinguish text from data such as images and audio: pixels and sound waves have measurable properties that can be encoded numerically, while text represents thoughts, ideas, and communication without a comparable direct translation of meaning. Explain that words can be turned into numbers and embeddings, but those encodings are approximations of meaning rather than direct measurements, and may themselves be produced using machine learning models. Describe the section’s aside that numbers also appear as text within language, so better mathematical ability may require a more meaningful way to encode them; converting written numbers into machine-readable numbers is not presented as a simple fix.

What does section 3.2.6, Token limits create bottlenecks, cover?

This section explains why LLM input-token limits are difficult to raise and how they affect response time, hardware requirements, and users of different languages. Explain that a model’s token limit is the maximum number of input tokens it can accept, and that raising it is constrained by GPU memory and speed as well as the quadratic growth of attention calculations. Describe how a higher token limit can improve available context yet make inference slower or exceed practical hardware capacity. Use the section’s examples of more timeouts after a GPT-3-to-GPT-4 upgrade and out-of-memory errors when running LLaMA on a consumer GPU. Explain that the same sentence can consume different numbers of tokens in different languages, leaving users with different amounts of input space under one token limit. Describe the section’s connection between this disparity, text encoding, and the greater byte lengths of many non-ASCII characters.

What does section 3.2.7, Hallucinations cause confusion, cover?

This section explains why plausible but false LLM outputs are difficult to spot and why production teams must prepare for them despite the lack of a reliable fix. Explain that a hallucination is an LLM output that sounds correct but is wrong or made up, and illustrate how a credible-looking book recommendation or URL can mislead a user. Describe how hallucinations are often expressed confidently and how training that rewards correct, direct-sounding answers can discourage expressions of uncertainty. Explain that hallucinations may have several causes rather than one known cause, so there is no good fix yet; state why production teams should be prepared to counter model inaccuracies and biases for users.

What does section 3.2.8, Bias and ethical considerations, cover?

This section explains why LLM operations must address both dangerous answers and biases learned from large, insufficiently filtered text datasets. Explain that an LLM can cause harm even when its answer is accurate, so preventing responses that encourage or enable serious harm is vital. Trace how biases in vast, largely uncurated text datasets are learned and then reproduced by LLMs as stereotypes or discriminatory language. Describe why cleaning is often avoided and how automated filtering could still help.

What does section 3.2.9, Security concerns, cover?

This section explains how LLMs can expose sensitive information through prompts or compromised model files, and why both risks need safeguards. Explain why harmful or sensitive material in an LLM’s training data, including company data used for finetuning, must be protected from inadvertent disclosure. Describe prompt injection as a user’s attempt to make an LLM ignore its instructions and produce disallowed content, including potentially revealing secrets or enabling unsafe actions. Distinguish a convincing-looking generated secret from a verified real one. Trace how a tampered pickle-serialized model can execute malicious code when deserialized for use, and state the section’s precautions of secure transfer and model-integrity verification.

What does section 3.2.10, Controlling costs, cover?

This section explains why LLM costs extend beyond hardware purchases and why variable token usage makes cost engineering essential. Explain how scarce, expensive GPUs, storage, computation, and energy raise LLM costs during training and inference. Describe how long deployment times and bursty traffic can lead teams to keep services running even when traffic is low. Describe costs beyond compute and serving: managing and storing training or finetuning data, maintaining models and security, addressing legal and regulatory concerns, and investing in research and development. Explain that many APIs charge by token, so longer context and more elaborate prompts cost more. Distinguish predictable input lengths from uncertain output lengths, and state why the section calls for cost engineering practices.

What should I know before reading chapter 3?

Summary (chapter 2): The preceding chapter’s takeaway about transformers and NLP.; Really big transformers (chapter 2): The discussion of very large transformers provides context for model size.; Embeddings (chapter 2): Introduces embeddings as a language-modeling technique.; Attention (chapter 2): The attention mechanism used to relate tokens in a sequence.; Decoders (chapter 2): Introduces the transformer decoder whose attention weights can be adapted.; Transformers (chapter 2): Introduces the transformer structure whose feed-forward layers MoE replaces.

Chapter 3: Large language model operations: Building a platform for LLMs

What you will learn in chapter 3 of LLMs in Production

The introduction explains why larger language models require scaled-up operations and infrastructure, then previews the chapter’s focus on deployment challenges, ways to reduce them, and supporting architecture.

After this chapter you will be able to

Model Size and Operational Scale. Explain that although bigger, linguistically informed transformer models can be more effective for NLP, their size creates operational and infrastructure challenges regardless of their linguistic efficacy.
LLMOps and MLOps relationship. Explain that the section treats LLMOps as MLOps scaled to handle LLMs, while questioning whether it is distinct enough to need its own term. Describe why a rudimentary MLOps setup may not be able to handle that scale.
MLOps lifecycle and practices. Describe MLOps as reliably and efficiently deploying and maintaining ML models in production across the whole lifecycle, from data acquisition and training through monitoring and termination. Name representative practices the section says this requires, such as orchestration, versioning, feedback loops, CI/CD, security, resource provisioning, or data governance.
Why LLMOps Is Distinct. Explain that LLMOps overlaps with MLOps but is not merely a fashionable new name: LLMs bring operational problems uncommon in traditional machine learning, largely because the models are so big.
Model size and download delays. Explain why downloading a large LLM must be planned for as part of production work: Bloom is about 330 GB, and downloading it took the author's team about an hour and a half.
Runtime downloads and restarts. Explain why practices that work for small models—downloading at run time and relying on quick application restarts—are poor fits for large LLMs. A shutdown may take seconds while redeployment can take much longer.
Scaling and release delays. Explain how long deployment times make scaling down before a traffic surge and handling bursty workloads difficult. Describe how slower rolling updates create CI/CD backlogs and make mistakes slower to notice and correct.
Model size and perceived speed. Explain how larger models tend to increase inference latency, and why streaming a response one word at a time can make a chatbot feel faster without eliminating the wait. Describe why that latency may make LLMs unsuitable for fast autocomplete or large-scale text-processing workflows.
Distribution and completion length costs. Explain that distributing an LLM across GPUs adds communication overhead, even when distribution is intended to improve latency. Describe how longer completions take longer to produce and how chain-of-thought prompting can trade additional response time for better results on logic or math questions.
GPU need for LLMs. Explain why LLM inference and training typically need GPUs rather than relying on CPUs: GPUs accelerate parallel processing and the linear algebra, matrix multiplication, and tensor operations underlying LLMs.
GPU infrastructure integration. Describe the operational changes introduced when an LLM service needs GPUs: architectures and orchestration tools often assume CPU and memory, so GPU support may require rebuilding containers and deploying new metrics systems.
GPU scarcity and placement. Explain how limited GPU supply complicates both on-premise and cloud deployments: on-premise servers may lack accessible GPUs, while cloud GPU types can be exhausted or unavailable in a region, potentially forcing deployment farther from users and increasing latency.
Approximate text encodings. Explain that words can be turned into numbers and embeddings, but those encodings are approximations of meaning rather than direct measurements, and may themselves be produced using machine learning models.
Text monitoring difficulties. Explain how the lack of direct measures for text makes operational questions difficult to define and answer, including detecting data drift, measuring correctness, and ensuring data cleanliness.
Sources of token limits. Explain that a model’s token limit is the maximum number of input tokens it can accept, and that raising it is constrained by GPU memory and speed as well as the quadratic growth of attention calculations.
Application tradeoffs of longer contexts. Describe how a higher token limit can improve available context yet make inference slower or exceed practical hardware capacity. Use the section’s examples of more timeouts after a GPT-3-to-GPT-4 upgrade and out-of-memory errors when running LLaMA on a consumer GPU.
Language differences in token usage. Explain that the same sentence can consume different numbers of tokens in different languages, leaving users with different amounts of input space under one token limit. Describe the section’s connection between this disparity, text encoding, and the greater byte lengths of many non-ASCII characters.
Plausible but false outputs. Explain that a hallucination is an LLM output that sounds correct but is wrong or made up, and illustrate how a credible-looking book recommendation or URL can mislead a user.
Uncertain causes and preparation. Explain that hallucinations may have several causes rather than one known cause, so there is no good fix yet; state why production teams should be prepared to counter model inaccuracies and biases for users.
Dangerous but correct answers. Explain that an LLM can cause harm even when its answer is accurate, so preventing responses that encourage or enable serious harm is vital.
Training data as bias source. Trace how biases in vast, largely uncurated text datasets are learned and then reproduced by LLMs as stereotypes or discriminatory language. Describe why cleaning is often avoided and how automated filtering could still help.
Prompt injection risks. Describe prompt injection as a user’s attempt to make an LLM ignore its instructions and produce disallowed content, including potentially revealing secrets or enabling unsafe actions. Distinguish a convincing-looking generated secret from a verified real one.
Pickle injection during transfer. Trace how a tampered pickle-serialized model can execute malicious code when deserialized for use, and state the section’s precautions of secure transfer and model-integrity verification.
Infrastructure and operating costs. Explain how scarce, expensive GPUs, storage, computation, and energy raise LLM costs during training and inference. Describe how long deployment times and bursty traffic can lead teams to keep services running even when traffic is low.
Token pricing and cost uncertainty. Explain that many APIs charge by token, so longer context and more elaborate prompts cost more. Distinguish predictable input lengths from uncertain output lengths, and state why the section calls for cost engineering practices.
Compression for smaller models. Explain that compression aims to make LLMs as small as possible by shrinking, trimming, or approximating them.
Distributed computing memory rationale. Explain why the section introduces distributed computing: LLMs are often too large to fit in a single GPU’s memory, so it is needed to make them run.
Quantization benefits and limits. Explain how representing model values with fewer bits reduces memory and computational costs, and why lower precision can eventually damage accuracy.
Structured and unstructured pruning. Distinguish removing unhelpful model components from zeroing individual parameters, including what each approach does to model size and inference latency.
Teacher student knowledge distillation. Explain how a large teacher model supplies examples to train a smaller student, and weigh the resulting deployment gains against training and maintenance costs.
Low rank approximation and LoRA. Describe how low-rank factorization replaces a large matrix with smaller factors, and distinguish that approximation from LoRA’s use of small trainable updates for finetuning.
Sparse mixture of experts. Explain how mixture-of-experts layers replace transformer feed-forward layers and route work to only a few experts, including the benefits and limitations of that structural change.
Data parallelism and Ray. Explain that data parallelism splits data across multiple copies of a model or pipeline, usually requiring each model copy to fit on one GPU. Describe how Ray distributes tasks across resources and can run LLM pipelines on cluster nodes with multiple GPUs.
Tensor parallelism tradeoffs. Explain how tensor parallelism divides expensive tensor operations across processors and combines their results. Distinguish column splitting, which replicates input and concatenates outputs, from row splitting, which divides input but must add outputs; describe why the chosen dimension depends on size, communication, computation, and hardware.
Pipeline model partitioning. Explain that pipeline parallelism places successive parts of a model on different GPUs and passes inputs between them in order. State why this can make a model that exceeds one GPU's memory runnable, unlike approaches that chiefly speed up work within a fitting model.
Pipeline bubbles and microbatches. Explain why sequential pipeline stages leave GPUs idle, and how feeding smaller microbatches lets stages overlap work. Describe the tradeoff that more microbatches reduce idle time but increase cached-activation memory, and how PipeDream starts backward passes early to release that cache.
Three dimensional parallelism placement. Describe 3D parallelism as the combined use of data, tensor, and pipeline parallelism. Explain why tensor-parallel GPUs should preferably be close together because of communication overhead, while pipeline stages are less costly to place across nodes.
Life cycle before tooling. Explain why the machine learning life cycle should guide infrastructure choices: infrastructure is how the work gets done, while the life cycle explains why it is needed. Describe why a bare-bones setup that works for smaller models may be inadequate for LLMs, including the difficulty of storing a model locally.
Infrastructure responsibilities by discipline. Distinguish the responsibilities of DevOps, DataOps, and MLOps: DevOps provides environments and foundational systems; DataOps handles data at rest and in motion; and MLOps supports models from creation through deprecation. Explain that DataOps builds on DevOps, while MLOps builds on both.
LLM platform components and destination. Describe the pictured platform as a combination of existing data and MLOps tools, with a vector database as the main new tool introduced for LLMs. State that monitoring is not pictured because it touches every piece, and that the components lead to a deployment service for running LLMs in production.
Core DataOps component roles. Describe how data stores hold managed and secured data, orchestrators automate multistep work across resources, and pipelines run jobs such as data preparation, scheduled training, and batch inference.
Shared experiment history. Explain why an experiment tracker records tests and results in a form others can find and examine, rather than leaving results in personal notebooks. Describe how that history helps a team investigate production problems instead of repeating mistakes.
Finetuned model lineage. Explain that an LLM finetuning experiment should record the checkpoint it started from and, if possible, link to the original training experiment. Describe how that link helps later investigators trace results and original training data, including when investigating bias.
Model evaluation comparison. Describe why LLM experiment trackers should make it easy to compare models across multiple evaluation metrics, even when the tracker does not prescribe which metrics to use. Give the section's example of ROUGE for text summarization.
Model storage and metadata. Explain that a model registry is primarily a place to store models, sometimes as simply as an object store or shared filesystem. Describe why it should also let teams link models to experiment-tracker information or tag models obtained from outside the company.
Model versions and replacement. Explain why a registry should version models: models eventually need replacing, and distinct versions make production experiments such as A/B tests and shadow tests easier.
Model access and promotion. Describe why registry access should be restricted: models can be valuable intellectual property, and only a team that understands them should be able to promote or demote them, avoiding damaging changes to production models.
LLM registry storage capacity. Explain why a registry's model-size limits and total storage capacity must be checked for LLMs. Describe how very large models, multiple checkpoints and versions, and reliability copies can quickly fill storage, even though storage is relatively cheap.
Feature store as showroom. Explain how a feature store provides a single source of truth for finding and using features, including their definitions, ownership, access, and model usage. Distinguish this role from simply storing data in another database.
Training-serving feature consistency. Explain how feature store abstractions reduce training-serving skew when a feature is easy to calculate after the fact but difficult to produce in real time. Describe how point-in-time retrieval returns the same result for a specified time even when the underlying feature changes.
External context for LLMs. Explain why the section argues that LLMs may still benefit from a feature store: the model is not fully self-sufficient, and supplying document embeddings as context can improve factual correctness and help avoid hallucinations.
Similarity Search Over Embeddings. Explain that vector databases store vectors with metadata, but their main value is finding nearby vectors when an exact ID or keyword is unavailable. Describe how this can retrieve relevant context for an LLM.
Feature Store and Vector Database. Distinguish a feature store from a vector database and explain why a feature store can be built on top of a vector database rather than replacing it. Describe how the combination helps manage changing data sources and embeddings while keeping search current.
Silent failure and data drift. Explain how a model can keep running while its results deteriorate, often because the data it encounters changes, and how monitoring helps teams detect the problem and adjust or retrain the model.
Monitoring measures and auditing. Describe the mix of ordinary service metrics, data-drift checks, and model-quality measures the section recommends. Explain why accuracy-style measures are hard to obtain at inference time and why an auditing system can help.
Limits of text monitoring. Explain why text drift is difficult to quantify: unique-token checks can spot new terms but not changed meanings, while embedding checks may create noisy alerts or be hard to interpret. Describe the trade-off of handcrafted monitoring rules and features.
GPU Workstations for Development. Explain why remote GPU-enabled workstations are a practical necessity for developing and troubleshooting LLMs, rather than a luxury.
Inference Memory Estimation. Apply the section’s inference shortcut: at FP16 or BF16, allow roughly two bytes per parameter, then add memory for an embedding model and the tokens being processed, particularly across batches.
Training Memory Budget. Distinguish training memory from inference memory using the section’s estimate of roughly 16 bytes per parameter for full-precision training plus gradients and optimizer tensors.
Deployment service role. Explain how a deployment service brings together earlier MLOps systems, defines the resources a model needs, and makes it available in production through an API or a batch or streaming pipeline.
Operational deployment bridge. Explain how the deployment service connects MLOps to a company's CI/CD and shipping pipelines, and configures an orchestrator, environment variables, and secrets so a model can run reliably. Relate this to the need to provision enough cluster resources for LLMs.

liveBook manning.com

Read chapter 3 on liveBook Get the book at manning.com

What it builds on

Summary (chapter 2) · The preceding chapter’s takeaway about transformers and NLP.
Really big transformers (chapter 2) · The discussion of very large transformers provides context for model size.
Embeddings (chapter 2) · Introduces embeddings as a language-modeling technique.
Attention (chapter 2) · The attention mechanism used to relate tokens in a sequence.
Decoders (chapter 2) · Introduces the transformer decoder whose attention weights can be adapted.
Transformers (chapter 2) · Introduces the transformer structure whose feed-forward layers MoE replaces.

Section by section

Introduction

Model Size and Operational Scale: Explain that although bigger, linguistically informed transformer models can be more effective for NLP, their size creates operational and infrastructure challenges regardless of their linguistic efficacy.
Why it matters: This explains why operating an LLM requires more preparation than choosing an effective model.
Chapter Operations Roadmap: Describe the chapter’s plan to identify LLM deployment challenges, consider ways to minimize them, and examine architecture and infrastructure that can help address them.
Why it matters: This roadmap connects the chapter’s challenges to its proposed operational responses.

Key terms: transformers, natural language processing (NLP), linguistically informed, operations, infrastructure, deployment challenges, large language model best practices, architecture

3.1 Introduction to large language model operations

Introduces LLMOps as MLOps scaled for large language models, then explains the production lifecycle and practices that MLOps covers.

LLMOps and MLOps relationship: Explain that the section treats LLMOps as MLOps scaled to handle LLMs, while questioning whether it is distinct enough to need its own term. Describe why a rudimentary MLOps setup may not be able to handle that scale.
Why it matters: This distinction frames the operational challenges discussed next without treating LLMOps as an entirely new discipline.
MLOps lifecycle and practices: Describe MLOps as reliably and efficiently deploying and maintaining ML models in production across the whole lifecycle, from data acquisition and training through monitoring and termination. Name representative practices the section says this requires, such as orchestration, versioning, feedback loops, CI/CD, security, resource provisioning, or data governance.
Why it matters: The full-lifecycle view explains why operating models involves more than deploying them once.

Key terms: LLMOps, MLOps, scaling, machine learning life cycle, workflow orchestration, versioning, feedback loops, CI/CD, security

3.2 Operations challenges with large language models

This introduction explains why LLMOps merits a distinction from MLOps and frames the operational challenges of deploying large models.

Why LLMOps Is Distinct: Explain that LLMOps overlaps with MLOps but is not merely a fashionable new name: LLMs bring operational problems uncommon in traditional machine learning, largely because the models are so big.
Why it matters: This distinction explains why deploying an LLM calls for specialized operational expertise.

Key terms: MLOps, LLMOps, Big Data, large models

3.2.1 Long download times

This section shows why the sheer size of LLMs makes downloading model files a production bottleneck, even when parallel downloads help.

Model size and download delays: Explain why downloading a large LLM must be planned for as part of production work: Bloom is about 330 GB, and downloading it took the author's team about an hour and a half.
Why it matters: A model cannot be put to work until its files arrive, so download time affects production plans.
Parallel downloads and realistic timing: Distinguish an ideal download-time estimate from a realistic one, and describe how splitting Bloom into smaller files permits parallel downloads without eliminating the delay.
Why it matters: Parallel downloads help, but an ideal network-speed calculation can understate the wait teams will face.

Key terms: Bloom, download times, model size, sharded, parallel, gigabit speed fiber, bandwidth, I/O operations

3.2.2 Longer deploy times

Large LLMs take long enough to deploy that teams must rethink runtime downloads, restarts, scaling, and release workflows.

Sources of deployment delay: Describe why deploying a large LLM can take much longer than downloading it: loading the model into GPU memory takes time, other deployment steps add time, and teams may have to wait for GPUs to become available.
Why it matters: These delays set the constraints for every deployment decision that follows.
Runtime downloads and restarts: Explain why practices that work for small models—downloading at run time and relying on quick application restarts—are poor fits for large LLMs. A shutdown may take seconds while redeployment can take much longer.
Why it matters: A restart is no longer a cheap way to recover from a problem.
Scaling and release delays: Explain how long deployment times make scaling down before a traffic surge and handling bursty workloads difficult. Describe how slower rolling updates create CI/CD backlogs and make mistakes slower to notice and correct.
Why it matters: Capacity and release plans must allow for the time it takes to bring an LLM service online.

Key terms: GPU memory, GPU shortages, Bloom, download the model at run time, restart, redeploy, semi-irreversible process, bursty workloads, CI/CD, rolling updates

3.2.3 Latency

This section explains why LLM inference latency can be a production bottleneck even when a streamed chatbot response feels fast.

Model size and perceived speed: Explain how larger models tend to increase inference latency, and why streaming a response one word at a time can make a chatbot feel faster without eliminating the wait. Describe why that latency may make LLMs unsuitable for fast autocomplete or large-scale text-processing workflows.
Why it matters: A smooth chatbot experience does not prove that an LLM is fast enough for another application.
Distribution and completion length costs: Explain that distributing an LLM across GPUs adds communication overhead, even when distribution is intended to improve latency. Describe how longer completions take longer to produce and how chain-of-thought prompting can trade additional response time for better results on logic or math questions.
Why it matters: Both the serving setup and the length of the requested answer affect how long a user waits.

Key terms: inference latency, parameters, streamed, autocomplete, multiple GPUs, communication overhead, completion length, chain of thought (CoT)

3.2.4 Managing GPUs

This section explains why LLMs need GPUs and why adding them to production systems creates tooling, provisioning, and latency challenges.

GPU need for LLMs: Explain why LLM inference and training typically need GPUs rather than relying on CPUs: GPUs accelerate parallel processing and the linear algebra, matrix multiplication, and tensor operations underlying LLMs.
Why it matters: The reason for using GPUs also explains why their availability affects LLM performance and deployment.
GPU infrastructure integration: Describe the operational changes introduced when an LLM service needs GPUs: architectures and orchestration tools often assume CPU and memory, so GPU support may require rebuilding containers and deploying new metrics systems.
Why it matters: Adding GPU compute can require changes to the platform, not just a different machine.
GPU scarcity and placement: Explain how limited GPU supply complicates both on-premise and cloud deployments: on-premise servers may lack accessible GPUs, while cloud GPU types can be exhausted or unavailable in a region, potentially forcing deployment farther from users and increasing latency.
Why it matters: GPU availability can determine where a service runs and how quickly users receive responses.

Key terms: GPUs, CPUs, parallel processing, linear algebra, matrix multiplications, tensor operations, system architectures, orchestrating tooling, Kubernetes, containers

3.2.5 Peculiarities of text data

The section explains why the meaning of text is harder to encode numerically than physical measurements, and how that difficulty complicates LLM operations.

Text versus physical measurements: Distinguish text from data such as images and audio: pixels and sound waves have measurable properties that can be encoded numerically, while text represents thoughts, ideas, and communication without a comparable direct translation of meaning.
Why it matters: This contrast explains why familiar approaches to measuring data do not transfer neatly to text.
Approximate text encodings: Explain that words can be turned into numbers and embeddings, but those encodings are approximations of meaning rather than direct measurements, and may themselves be produced using machine learning models.
Why it matters: Treating an embedding as an approximation helps explain the limits of what numerical text representations can capture.
Numbers within language: Describe the section’s aside that numbers also appear as text within language, so better mathematical ability may require a more meaningful way to encode them; converting written numbers into machine-readable numbers is not presented as a simple fix.
Why it matters: The example shows that even apparently numerical text poses a representation problem.
Text monitoring difficulties: Explain how the lack of direct measures for text makes operational questions difficult to define and answer, including detecting data drift, measuring correctness, and ensuring data cleanliness.
Why it matters: These are practical monitoring challenges that follow from the difficulty of measuring text meaning.

Key terms: text data, qualitative, pixels, RGB values, decibels, hertz, embeddings, encode text, machine learning models, numbers

3.2.6 Token limits create bottlenecks

This section explains why LLM input-token limits are difficult to raise and how they affect response time, hardware requirements, and users of different languages.

Sources of token limits: Explain that a model’s token limit is the maximum number of input tokens it can accept, and that raising it is constrained by GPU memory and speed as well as the quadratic growth of attention calculations.
Why it matters: A larger context window is useful, but supporting it requires more than simply assigning more memory.
Application tradeoffs of longer contexts: Describe how a higher token limit can improve available context yet make inference slower or exceed practical hardware capacity. Use the section’s examples of more timeouts after a GPT-3-to-GPT-4 upgrade and out-of-memory errors when running LLaMA on a consumer GPU.
Why it matters: An application may benefit more from a timely answer than from extra context that causes a timeout or crash.
Language differences in token usage: Explain that the same sentence can consume different numbers of tokens in different languages, leaving users with different amounts of input space under one token limit. Describe the section’s connection between this disparity, text encoding, and the greater byte lengths of many non-ASCII characters.
Why it matters: A chatbot with one token limit may offer substantially less input flexibility to some language users than to English users.

Key terms: token limit, context, GPU memory, attention, quadratic, inference times, timeout error rate, out-of-memory (OOM) errors, LLaMA, tokens per character

3.2.7 Hallucinations cause confusion

This section explains why plausible but false LLM outputs are difficult to spot and why production teams must prepare for them despite the lack of a reliable fix.

Plausible but false outputs: Explain that a hallucination is an LLM output that sounds correct but is wrong or made up, and illustrate how a credible-looking book recommendation or URL can mislead a user.
Why it matters: A convincing format is not evidence that an LLM's answer is accurate.
Confidence despite uncertainty: Describe how hallucinations are often expressed confidently and how training that rewards correct, direct-sounding answers can discourage expressions of uncertainty.
Why it matters: Users may mistake a confident tone for a reliable answer.
Uncertain causes and preparation: Explain that hallucinations may have several causes rather than one known cause, so there is no good fix yet; state why production teams should be prepared to counter model inaccuracies and biases for users.
Why it matters: A production team cannot assume hallucinations will disappear through a single fix.

Key terms: hallucinations, uncertainty, rewards, inaccuracies, biases

3.2.8 Bias and ethical considerations

This section explains why LLM operations must address both dangerous answers and biases learned from large, insufficiently filtered text datasets.

Dangerous but correct answers: Explain that an LLM can cause harm even when its answer is accurate, so preventing responses that encourage or enable serious harm is vital.
Why it matters: Checking whether an answer is factually correct is not enough to judge whether it is safe.
Training data as bias source: Trace how biases in vast, largely uncurated text datasets are learned and then reproduced by LLMs as stereotypes or discriminatory language. Describe why cleaning is often avoided and how automated filtering could still help.
Why it matters: Bias in model outputs is tied to choices about the data used to train the model.

Key terms: bias and ethical considerations, text data, bias, curated, filtered, regular expressions, implicit human biases

3.2.9 Security concerns

This section explains how LLMs can expose sensitive information through prompts or compromised model files, and why both risks need safeguards.

Sensitive data exposure: Explain why harmful or sensitive material in an LLM’s training data, including company data used for finetuning, must be protected from inadvertent disclosure.
Why it matters: A model can expose information that users were never meant to see.
Prompt injection risks: Describe prompt injection as a user’s attempt to make an LLM ignore its instructions and produce disallowed content, including potentially revealing secrets or enabling unsafe actions. Distinguish a convincing-looking generated secret from a verified real one.
Why it matters: An apparent jailbreak may create a real security breach—or merely produce a plausible hallucination.
Pickle injection during transfer: Trace how a tampered pickle-serialized model can execute malicious code when deserialized for use, and state the section’s precautions of secure transfer and model-integrity verification.
Why it matters: A compromised model file can give an attacker access to the system running it.

Key terms: training corpus, finetuning, sensitive data, prompt injection, jailbreak, hallucinations, API keys, environment variables, pickle, pickle injections

3.2.10 Controlling costs

This section explains why LLM costs extend beyond hardware purchases and why variable token usage makes cost engineering essential.

Infrastructure and operating costs: Explain how scarce, expensive GPUs, storage, computation, and energy raise LLM costs during training and inference. Describe how long deployment times and bursty traffic can lead teams to keep services running even when traffic is low.
Why it matters: These costs can accumulate even when a service is handling few requests.
Broader lifecycle expenses: Describe costs beyond compute and serving: managing and storing training or finetuning data, maintaining models and security, addressing legal and regulatory concerns, and investing in research and development.
Why it matters: A budget focused only on GPUs misses substantial ongoing expenses.
Token pricing and cost uncertainty: Explain that many APIs charge by token, so longer context and more elaborate prompts cost more. Distinguish predictable input lengths from uncertain output lengths, and state why the section calls for cost engineering practices.
Why it matters: Variable output length makes API spending harder to forecast and govern.

Key terms: infrastructure costs, GPUs, training, inference, operational costs, data, finetune, model updates, security measures, compliance

3.3 LLMOps essentials

This section introduces compression and distributed computing as responses to LLM operations challenges, then points ahead to the infrastructure and tooling that support them.

Compression for smaller models: Explain that compression aims to make LLMs as small as possible by shrinking, trimming, or approximating them.
Why it matters: Reducing model size is one of the first practices the chapter considers for overcoming LLM operations challenges.
Distributed computing memory rationale: Explain why the section introduces distributed computing: LLMs are often too large to fit in a single GPU’s memory, so it is needed to make them run.
Why it matters: The single-GPU memory limit explains why distributed computing is an LLMOps essential.

Key terms: compression, shrinking, trimming, approximating, distributed computing, single GPU’s memory

3.3.1 Compression

This section compares ways to reduce an LLM’s memory or computation costs while managing the resulting accuracy, training, and deployment tradeoffs.

Quantization benefits and limits: Explain how representing model values with fewer bits reduces memory and computational costs, and why lower precision can eventually damage accuracy.
Why it matters: Quantization is an accessible way to make large models cheaper to deploy, but its accuracy tradeoff matters.
Quantization method tradeoffs: Distinguish straightforward post-training quantization from choices that manage clipping, rounding, and accuracy loss, including dynamic ranges and quantization-aware training.
Why it matters: These choices explain why an easy size reduction may work immediately or require more effort to preserve accuracy.
Structured and unstructured pruning: Distinguish removing unhelpful model components from zeroing individual parameters, including what each approach does to model size and inference latency.
Why it matters: A smaller model does not necessarily perform fewer computations, so the kind of pruning matters for deployment.
Teacher student knowledge distillation: Explain how a large teacher model supplies examples to train a smaller student, and weigh the resulting deployment gains against training and maintenance costs.
Why it matters: Distillation can deliver a genuinely smaller, faster model, but creating and keeping it current is a separate project.
Low rank approximation and LoRA: Describe how low-rank factorization replaces a large matrix with smaller factors, and distinguish that approximation from LoRA’s use of small trainable updates for finetuning.
Why it matters: The section’s practical use of low-rank methods is efficient adaptation, not an assured production-ready compression of the original weights.
Sparse mixture of experts: Explain how mixture-of-experts layers replace transformer feed-forward layers and route work to only a few experts, including the benefits and limitations of that structural change.
Why it matters: MoE targets inference computation through selective activation, which is different from merely reducing stored parameter counts.

Key terms: quantization, FP32, FP16, BF16, INT8, INT2, post-training static quantization, PTQ, dynamic quantization, clipping errors

3.3.2 Distributed computing

Explains how data, tensor, sequence, and pipeline parallelism distribute LLM workloads, and how combining them affects memory use, communication, and GPU utilization.

Data parallelism and Ray: Explain that data parallelism splits data across multiple copies of a model or pipeline, usually requiring each model copy to fit on one GPU. Describe how Ray distributes tasks across resources and can run LLM pipelines on cluster nodes with multiple GPUs.
Why it matters: Data parallelism increases throughput, but it does not by itself solve the problem of a model that cannot fit on one GPU.
Tensor parallelism tradeoffs: Explain how tensor parallelism divides expensive tensor operations across processors and combines their results. Distinguish column splitting, which replicates input and concatenates outputs, from row splitting, which divides input but must add outputs; describe why the chosen dimension depends on size, communication, computation, and hardware.
Why it matters: The split that makes a large operation feasible can also determine its communication and computation costs.
Sequence and tensor parallelism: Explain that tensor parallelism targets computation-heavy transformer layers such as MLP and attention, while sequence parallelism partitions activations along the sequence dimension to reduce redundant storage in other blocks, including normalization and dropout. Describe why combining them improves parallel coverage and activation-memory use.
Why it matters: Splitting expensive calculations is not enough if the remaining blocks still consume substantial activation memory.
Pipeline model partitioning: Explain that pipeline parallelism places successive parts of a model on different GPUs and passes inputs between them in order. State why this can make a model that exceeds one GPU's memory runnable, unlike approaches that chiefly speed up work within a fitting model.
Why it matters: For an oversized model, distributing its stages can be a prerequisite for running it at all.
Pipeline bubbles and microbatches: Explain why sequential pipeline stages leave GPUs idle, and how feeding smaller microbatches lets stages overlap work. Describe the tradeoff that more microbatches reduce idle time but increase cached-activation memory, and how PipeDream starts backward passes early to release that cache.
Why it matters: A pipeline may fit the model yet waste expensive GPUs unless its idle time and memory demands are managed.
Three dimensional parallelism placement: Describe 3D parallelism as the combined use of data, tensor, and pipeline parallelism. Explain why tensor-parallel GPUs should preferably be close together because of communication overhead, while pipeline stages are less costly to place across nodes.
Why it matters: Combining the methods can outperform using them separately, but placement affects how much benefit the system gets.

Key terms: data parallelism, DistributedDataParallel, Ray, tasks, actors, tensor parallelism, column parallelism, row parallelism, MLP, attention

3.4 Infrastructure, GPUs, and Vector Databases

This section explains why LLM infrastructure must support the machine learning life cycle and maps the DevOps, DataOps, and MLOps components that lead to production deployment.

Life cycle before tooling: Explain why the machine learning life cycle should guide infrastructure choices: infrastructure is how the work gets done, while the life cycle explains why it is needed. Describe why a bare-bones setup that works for smaller models may be inadequate for LLMs, including the difficulty of storing a model locally.
Why it matters: Starting with the life cycle helps you choose infrastructure for a purpose rather than collecting tools.
Infrastructure responsibilities by discipline: Distinguish the responsibilities of DevOps, DataOps, and MLOps: DevOps provides environments and foundational systems; DataOps handles data at rest and in motion; and MLOps supports models from creation through deprecation. Explain that DataOps builds on DevOps, while MLOps builds on both.
Why it matters: The boundaries show how the platform's parts depend on one another.
LLM platform components and destination: Describe the pictured platform as a combination of existing data and MLOps tools, with a vector database as the main new tool introduced for LLMs. State that monitoring is not pictured because it touches every piece, and that the components lead to a deployment service for running LLMs in production.
Why it matters: This overview connects the individual tools to the production service they are meant to support.

Key terms: machine learning life cycle, infrastructure, LLMs, DevOps, DataOps, MLOps, data at rest, data in motion, vector database, monitoring system

3.4.1 Data infrastructure

This section explains the DataOps infrastructure beneath MLOps, the roles of its main components, and two additional infrastructure considerations.

Operations infrastructure layers: Explain that MLOps depends on DataOps infrastructure, which in turn builds on DevOps.
Why it matters: This places LLM operations within the broader systems that support it.
Core DataOps component roles: Describe how data stores hold managed and secured data, orchestrators automate multistep work across resources, and pipelines run jobs such as data preparation, scheduled training, and batch inference.
Why it matters: These roles explain how data and ML work gets organized and run.
Container registry and consistency: Explain why a container registry is an additional infrastructure component: pipelines and services run in containers to maintain consistency.
Why it matters: Consistent execution matters when jobs and services run across operational systems.
Text streaming requirements: Distinguish streaming integration from an assumption that every text task requires real-time processing; explain that even captioning or translation can sometimes use pseudo-real-time processing without harming the experience, depending on the task.
Why it matters: The required degree of real-time processing depends on the task and user experience.

Key terms: MLOps, DataOps, DevOps, data store, orchestrator, pipelines, batch inference, container registry, containers, consistency

3.4.2 Experiment trackers

Experiment trackers make LLM experiments searchable and useful to a team, provided they record model lineage and support meaningful evaluation.

Shared experiment history: Explain why an experiment tracker records tests and results in a form others can find and examine, rather than leaving results in personal notebooks. Describe how that history helps a team investigate production problems instead of repeating mistakes.
Why it matters: A searchable record lets the next person learn from an experiment, especially when a deployed model fails.
Finetuned model lineage: Explain that an LLM finetuning experiment should record the checkpoint it started from and, if possible, link to the original training experiment. Describe how that link helps later investigators trace results and original training data, including when investigating bias.
Why it matters: Recording the starting model makes a finetuning result traceable rather than an isolated run.
Model evaluation comparison: Describe why LLM experiment trackers should make it easy to compare models across multiple evaluation metrics, even when the tracker does not prescribe which metrics to use. Give the section's example of ROUGE for text summarization.
Why it matters: Comparing recorded metrics helps a team decide between models when no single simple score tells the whole story.
Tracker ecosystem and integrations: State that MLFlow is a popular open-source tracker, with CometML and Weights & Biases as paid alternatives, and describe examples of LLM-specific integrations or tools to look for. Note that their feature sets are still developing.
Why it matters: Knowing which capabilities to check is more useful than assuming every tracker currently has the same LLM features.

Key terms: experiment trackers, tests and results, MLOps, finetuning, checkpoint, original training experiment, training data, evaluation metric tooling, evaluation metrics, ROUGE

3.4.3 Model registry

This section explains what a model registry needs to do and why LLM size makes storage capacity especially important.

Model storage and metadata: Explain that a model registry is primarily a place to store models, sometimes as simply as an object store or shared filesystem. Describe why it should also let teams link models to experiment-tracker information or tag models obtained from outside the company.
Why it matters: A storage location is useful only if the team can identify a model and find the information associated with it.
Model versions and replacement: Explain why a registry should version models: models eventually need replacing, and distinct versions make production experiments such as A/B tests and shadow tests easier.
Why it matters: Versioning makes model replacement and comparisons in production easier to manage.
Model access and promotion: Describe why registry access should be restricted: models can be valuable intellectual property, and only a team that understands them should be able to promote or demote them, avoiding damaging changes to production models.
Why it matters: Access controls protect both the models themselves and decisions about which models run in production.
LLM registry storage capacity: Explain why a registry's model-size limits and total storage capacity must be checked for LLMs. Describe how very large models, multiple checkpoints and versions, and reliability copies can quickly fill storage, even though storage is relatively cheap.
Why it matters: A registry that works for smaller models may reject an LLM or run out of space as copies accumulate.

Key terms: model registry, object store, shared filesystem, metadata, experiment tracker, versioning, A/B testing, shadow tests, access, promoting

3.4.4 Feature stores

This section explains how feature stores make features discoverable and consistent across training and serving, and why they can help LLMs use external context.

Feature store as showroom: Explain how a feature store provides a single source of truth for finding and using features, including their definitions, ownership, access, and model usage. Distinguish this role from simply storing data in another database.
Why it matters: A feature store helps teams find trustworthy features without having to navigate raw data stores on their own.
Training-serving feature consistency: Explain how feature store abstractions reduce training-serving skew when a feature is easy to calculate after the fact but difficult to produce in real time. Describe how point-in-time retrieval returns the same result for a specified time even when the underlying feature changes.
Why it matters: Consistent feature values help models behave predictably when they move from training to production.
External context for LLMs: Explain why the section argues that LLMs may still benefit from a feature store: the model is not fully self-sufficient, and supplying document embeddings as context can improve factual correctness and help avoid hallucinations.
Why it matters: Even a large model may need relevant external context to give a reliable answer.

Key terms: feature store, single source of truth, collaboration, documentation, versioning, training-serving skew, online feature serving, point-in-time retrievals, LLMs, embeddings

3.4.5 Vector databases

The section explains how vector databases retrieve similar embeddings, how they work with feature stores, and what to consider when choosing one for an LLM application.

Similarity Search Over Embeddings: Explain that vector databases store vectors with metadata, but their main value is finding nearby vectors when an exact ID or keyword is unavailable. Describe how this can retrieve relevant context for an LLM.
Why it matters: Similarity search lets an application find useful material even when a query does not use the material’s exact words.
Feature Store and Vector Database: Distinguish a feature store from a vector database and explain why a feature store can be built on top of a vector database rather than replacing it. Describe how the combination helps manage changing data sources and embeddings while keeping search current.
Why it matters: The two tools address different problems, so using both can make an embedding search system easier to maintain.
Vector Database Selection Criteria: Describe the section’s priorities for choosing a vector database: price and scalability first, then supported similarity measures, indexing methods, and configurable search settings. Explain that tuning those settings helps balance query latency against search-result accuracy for a particular workload.
Why it matters: A vector database choice has to fit both the application’s budget and the speed and quality its searches require.
Vector Search Product Options: Distinguish the starting options described in the section: Pinecone as a fully managed offering, Milvus as an open-source option that can be self-deployed or obtained as a managed offering, and existing databases such as Redis or Elastic as possible quick starts with simpler vector-search features.
Why it matters: The available options differ in how much infrastructure you manage and whether you can build on tools you already use.

Key terms: vector databases, embeddings, metadata, nearest neighbors, similarity searches, cosine similarity, feature store, vector DB, embedding models, data sources

3.4.6 Monitoring system

This section explains why LLM monitoring matters, what to measure, and why changes in language make monitoring difficult.

Silent failure and data drift: Explain how a model can keep running while its results deteriorate, often because the data it encounters changes, and how monitoring helps teams detect the problem and adjust or retrain the model.
Why it matters: A healthy-looking service can still be giving users increasingly poor answers.
Monitoring measures and auditing: Describe the mix of ordinary service metrics, data-drift checks, and model-quality measures the section recommends. Explain why accuracy-style measures are hard to obtain at inference time and why an auditing system can help.
Why it matters: No single metric tells a team whether both the service and its answers are healthy.
Limits of text monitoring: Explain why text drift is difficult to quantify: unique-token checks can spot new terms but not changed meanings, while embedding checks may create noisy alerts or be hard to interpret. Describe the trade-off of handcrafted monitoring rules and features.
Why it matters: A text-monitoring signal can miss meaningful changes or raise alarms that are hard to investigate.
Monitoring before production: Describe why teams may add monitoring late, but should not wait too long or assume it begins only after deployment; monitoring can also support data governance and compliance in training and data pipelines.
Why it matters: Early pipeline checks can reveal problems before a production model goes unnoticed for too long.

Key terms: fail silently, data drift, monitoring system, memory and CPU utilization, latency, queries per second, status codes, error rates, missing values, unique tokens

3.4.7 GPU-enabled workstations

This section explains why LLM teams need GPU-enabled workstations and how to estimate GPU memory for inference and training.

GPU Workstations for Development: Explain why remote GPU-enabled workstations are a practical necessity for developing and troubleshooting LLMs, rather than a luxury.
Why it matters: Engineers need somewhere they can load and investigate models that no longer fit on a typical laptop.
GPU Memory Tiers: Describe the GPU memory options the section highlights, from cost-effective 16 GB workhorses through higher-memory GPUs suited to LLMs, and relate the choice to model fit and availability.
Why it matters: A GPU is useful only if it has enough available memory for the intended model and workload.
Inference Memory Estimation: Apply the section’s inference shortcut: at FP16 or BF16, allow roughly two bytes per parameter, then add memory for an embedding model and the tokens being processed, particularly across batches.
Why it matters: Model weights alone understate how much GPU memory an inference workload needs.
Training Memory Budget: Distinguish training memory from inference memory using the section’s estimate of roughly 16 bytes per parameter for full-precision training plus gradients and optimizer tensors.
Why it matters: Sizing hardware from an inference estimate would substantially underbudget a training job.

Key terms: GPU-enabled workstations, remote workstations, on-premise, T4, V100, A100, H100, L4 Tensor Core GPU, GPU memory, FP16

3.4.8 Deployment service

The section explains how a deployment service turns an LLM model and its supporting infrastructure into a production service, from serving interfaces to operational resources and release pipelines.

Deployment service role: Explain how a deployment service brings together earlier MLOps systems, defines the resources a model needs, and makes it available in production through an API or a batch or streaming pipeline.
Why it matters: The deployment service is the point where the other platform components become a working production system.
Standard and custom serving interfaces: Describe how serving tools can supply boilerplate and a standard interface, typically the KServe V2 Inference Protocol, and when a team might instead use FastAPI for customization while retaining automation where possible.
Why it matters: A standard interface reduces serving work, while customization remains an option when its value justifies it.
Operational deployment bridge: Explain how the deployment service connects MLOps to a company's CI/CD and shipping pipelines, and configures an orchestrator, environment variables, and secrets so a model can run reliably. Relate this to the need to provision enough cluster resources for LLMs.
Why it matters: A working model interface is not enough without release controls and sufficient production resources.

Key terms: deployment service, MLOps, REST, gRPC, batch or streaming pipeline, NVIDIA Triton Inference Service, MLServer, Seldon, BentoML, KServe V2 Inference Protocol

3.5 Summary

The chapter summary connects LLM operational challenges to model compression, parallelism, and infrastructure designed for LLM workloads.

Where this is used later

LLM storage strategies (chapter 6) · Provides context for considering how LLM files are stored and made available to a service.
Rolling updates (chapter 6) · Rolling updates must account for slow replacement deployments.
Autoscaling (chapter 6) · Autoscaling decisions must account for slow capacity startup.
Troubleshooting poor latency (chapter 6) · Troubleshooting poor latency requires distinguishing perceived responsiveness from the time inference takes.
Model compilation (chapter 6) · Latency as an operational challenge for LLMs.
Adaptive request batching (chapter 6) · Latency is an operational concern for LLM services.
Multi-GPU environments (chapter 5) · Multi-GPU training builds on the need for GPU compute.
Provisioning clusters (chapter 6) · Cluster provisioning must account for GPU resources.

Concepts explained in this chapter

pruningStructured pruning removes whole structural components of a neural network, such as filters, channels, or layers.
distillationDistillation is a model-training technique in which a student model learns to reproduce behavior or outputs from a teacher model.
low-rank approximationA method that represents a matrix or parameter structure with a lower-rank approximation to reduce storage or computation.
compressionCompression is the reduction of a model's size and computational requirements while retaining acceptable behavior and accuracy.
low-rank adaptationLow-rank adaptation is a parameter-efficient fine-tuning method that adapts a pretrained model through trainable low-rank updates while leaving most original parameters fixed.
mixture of expertsA model composed of multiple specialized expert subnetworks whose participation is selected or weighted for each input.
quantizationQuantization is the conversion of model parameters or computations from higher-precision numerical representations to lower-precision representations.
data parallelData parallelism is a parallel-computing technique that processes separate portions of data concurrently with multiple copies of a model or pipeline.
n-dimensional parallelism3D parallelism is a distributed-training strategy that combines data parallelism, tensor parallelism, and pipeline parallelism.
pipeline parallelismPipeline parallelism is a distributed-computing technique that partitions a model into sequential stages placed on different devices.
tensor parallelismTensor parallelism is a distributed-computing technique that partitions tensor computations across multiple processors and combines their results.
feature storeA feature store is a system or abstraction for managing, documenting, versioning, governing, and serving machine-learning features for training and production.
vector databaseA vector database is a database optimized for storing and retrieving numerical vector representations using similarity-based search.
distributed systemDistributed computing is a computing technique that divides workloads across multiple networked devices or nodes for concurrent execution.
RayRay is an open-source framework for executing distributed and parallel Python applications across clusters.
batchingA microbatch is a subdivision of a larger batch that is processed as one unit within a parallel training pipeline.
idle percentageIdle percentage is the proportion of pipeline execution time during which processing devices are not performing useful computation.
sequence parallelismSequence parallelism is a distributed-computing technique that partitions neural-network activations along the sequence dimension to reduce redundant storage.
experiment trackingAn experiment tracker is a tool that records machine-learning experiments, configurations, artifacts, and results so they can be inspected and compared.
mlopsMachine learning operations is the discipline of reliably deploying, monitoring, maintaining, and governing machine-learning systems throughout their life cycle.
monitoringA monitoring system observes a machine-learning system's data, resources, behavior, and outputs to detect degradation, failures, and operational issues.
deployment serviceA deployment service is a software service that configures, provisions, and exposes trained machine-learning models for inference in an operational environment.
model registryA model registry is a system for storing, versioning, annotating, and controlling access to trained machine-learning models.
GPUA GPU is a processor designed for highly parallel numerical computation, commonly used to accelerate machine-learning workloads.
driftData drift is a change over time in the statistical properties or distribution of data presented to a machine-learning system.
DataOpsDataOps is a discipline for managing the development, movement, quality, and operation of data systems and pipelines.
LLMOpsLLMOps is the discipline of developing, deploying, monitoring, and maintaining applications and systems built around large language models.
parameterA model parameter is a learned numerical value that determines part of a machine-learning model's computation.
similarity searchSemantic search retrieves information according to the meaning represented by queries and documents rather than relying only on exact lexical matches.
DevOpsDevOps is the set of practices that integrates software development and IT operations to automate and improve software delivery and operation.
infrastructureInfrastructure is the hardware, software, networking, and operational resources that support computing systems.
machine learning workflowA machine learning life cycle is the sequence of activities involved in creating, deploying, operating, and retiring machine-learning models.
attentionAn attention mechanism dynamically weights relationships between elements of an input sequence according to their relevance to a current computation.
token limitA token limit is a maximum number of tokens that a language-model API or computation accepts or processes for a request.
LlamaLLaMA is a family of autoregressive large language models released by Meta for research and development.
batch sizeBatch size is the number of input examples or inference requests processed together in one computation batch.
FastAPIFastAPI is a Python web framework for building HTTP APIs with type-based request validation and automatic interface documentation.
adversarial inputThe study and use of inputs or models designed to manipulate, evade, or expose weaknesses in machine-learning systems.
prompt injectionPrompt injection is an attack in which crafted input attempts to manipulate a language model into ignoring intended instructions or revealing unintended information.
versioningVersioning is the practice of recording and managing distinct revisions of software, data, models, or configuration.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.