What you will learn in LLMs in Production

From language models to successful products

by Christopher Brousseau and Matthew Sharp
Foreword by Joe Reis

Read it on liveBook Get the book at manning.com

Chapter by chapter

Chapter 1: Generative AI: Why large language models have captured attention

The chapter introduction compares the excitement around LLMs to the arrival of the printing press and previews questions the chapter will address.

1.1 Large language models accelerating communication · 1.2 Navigating the build-and-buy decision with LLMs · 1.2.1 Buying: The beaten path · 1.2.2 Building: The path less traveled · 1.2.3 A word of warning: Embrace the future now · 1.3 Debunking myths · 1.4 Summary

What you learn in chapter 1 →

Chapter 2: Large language models: A deep dive into language modeling

This introduction maps the chapter’s path from linguistic foundations and earlier language models to transformers and LLMs, and explains why that background matters in production.

2.1 Language modeling · 2.1.1 Linguistic features · 2.1.2 Semiotics · 2.1.3 Multilingual NLP · 2.2 Language modeling techniques · 2.2.1 N-gram and corpus-based techniques · 2.2.2 Bayesian techniques · 2.2.3 Markov chains · 2.2.4 Continuous language modeling · 2.2.5 Embeddings · 2.2.6 Multilayer perceptrons · 2.2.7 Recurrent neural networks and long short-term memory networks · 2.2.8 Attention · 2.3 Attention is all you need · 2.3.1 Encoders · 2.3.2 Decoders · 2.3.3 Transformers · 2.4 Really big transformers · 2.5 Summary

What you learn in chapter 2 →

Chapter 3: Large language model operations: Building a platform for LLMs

The introduction explains why larger language models require scaled-up operations and infrastructure, then previews the chapter’s focus on deployment challenges, ways to reduce them, and supporting architecture.

3.1 Introduction to large language model operations · 3.2 Operations challenges with large language models · 3.2.1 Long download times · 3.2.2 Longer deploy times · 3.2.3 Latency · 3.2.4 Managing GPUs · 3.2.5 Peculiarities of text data · 3.2.6 Token limits create bottlenecks · 3.2.7 Hallucinations cause confusion · 3.2.8 Bias and ethical considerations · 3.2.9 Security concerns · 3.2.10 Controlling costs · 3.3 LLMOps essentials · 3.3.1 Compression · 3.3.2 Distributed computing · 3.4 Infrastructure, GPUs, and Vector Databases · 3.4.1 Data infrastructure · 3.4.2 Experiment trackers · 3.4.3 Model registry · 3.4.4 Feature stores · 3.4.5 Vector databases · 3.4.6 Monitoring system · 3.4.7 GPU-enabled workstations · 3.4.8 Deployment service · 3.5 Summary

What you learn in chapter 3 →

Chapter 4: Data engineering for large language models: Setting up for success

The chapter introduces the model and data assets to prepare before training or finetuning an LLM, then previews how it will evaluate models and build a reusable dataset.

4.1 Models are the foundation · 4.1.1 GPT · 4.1.2 BLOOM · 4.1.3 LLaMA · 4.1.4 Wizard · 4.1.5 Falcon · 4.1.6 Vicuna · 4.1.7 Dolly · 4.1.8 OpenChat · 4.2 Evaluating LLMs · 4.2.1 Metrics for evaluating text · 4.2.2 Industry benchmarks · 4.2.3 Responsible AI benchmarks · 4.2.4 Developing your own benchmark · 4.2.5 Evaluating code generators · 4.2.6 Evaluating model parameters · 4.3 Data for LLMs · 4.3.1 Datasets you should know · 4.3.2 Data cleaning and preparation · 4.4 Text processors · 4.4.1 Tokenization · 4.4.2 Embeddings · 4.5 Preparing a Slack dataset · 4.6 Summary

What you learn in chapter 4 →

Chapter 5: Training large language models: How to generate the generator

The chapter introduces training an LLM on a prepared dataset and previews the environment, training methods, and practical advice it will cover.

5.1 Multi-GPU environments · 5.1.1 Setting up · 5.1.2 Libraries · 5.2 Basic training techniques · 5.2.1 From scratch · 5.2.2 Transfer learning (finetuning) · 5.2.3 Prompting · 5.3 Advanced training techniques · 5.3.1 Prompt tuning · 5.3.2 Finetuning with knowledge distillation · 5.3.3 Reinforcement learning with human feedback · 5.3.4 Mixture of experts · 5.3.5 LoRA and PEFT · 5.4 Training tips and tricks · 5.4.1 Training data size notes · 5.4.2 Efficient training · 5.4.3 Local minima traps · 5.4.4 Hyperparameter tuning tips · 5.4.5 A note on operating systems · 5.4.6 Activation function advice · 5.5 Summary

What you learn in chapter 5 →

Chapter 6: Large language model services: A practical guide

This introduction defines production as putting an LLM to work for real users and previews the chapter’s path from serving and cloud deployment to production challenges and edge devices.

6.1 Creating an LLM service with RAG and more · 6.1.1 Model compilation · 6.1.2 LLM storage strategies · 6.1.3 Adaptive request batching · 6.1.4 Flow control · 6.1.5 Streaming responses · 6.1.6 Feature store · 6.1.7 Retrieval-augmented generation · 6.1.8 LLM service libraries · 6.2 Setting up infrastructure · 6.2.1 Provisioning clusters · 6.2.2 Autoscaling · 6.2.3 Rolling updates · 6.2.4 Inference graphs · 6.2.5 Monitoring · 6.3 Cost engineering, security, latency, and other production challenges · 6.3.1 Model updates and retraining · 6.3.2 Load testing · 6.3.3 Troubleshooting poor latency · 6.3.4 Resource management · 6.3.5 Cost engineering · 6.3.6 Security · 6.4 Deploying to the edge · 6.5 Summary

What you learn in chapter 6 →

Chapter 7: Prompt engineering: Becoming an LLM whisperer

The introduction explains why prompting lets one general-purpose LLM serve many tasks and previews the chapter’s approach to prompt engineering.

7.1 Prompting your model · 7.1.1 Few-shot prompting · 7.1.2 One-shot prompting · 7.1.3 Zero-shot prompting · 7.2 Prompt engineering basics · 7.2.1 Anatomy of a prompt · 7.2.2 Prompting hyperparameters · 7.2.3 Scrounging the training data · 7.3 Tooling for structured outputs, workflows, and more · 7.3.1 LangChain · 7.3.2 Guidance · 7.3.3 DSPy · 7.3.4 Other tooling is available but … · 7.4 Advanced prompt engineering techniques · 7.4.1 Giving LLMs tools · 7.4.2 ReAct · 7.5 Summary

What you learn in chapter 7 →

Chapter 8: Applications and Agents: Building an interactive experience

The chapter introduces the shift from building and deploying LLMs to creating user-facing applications, including edge applications and agents.

8.1 Building an application with RAG · 8.1.1 Streaming on the frontend · 8.1.2 Keeping a history · 8.1.3 Chatbot interaction features · 8.1.4 Token counting · 8.1.5 RAG applied · 8.2 Edge applications · 8.3 LLM agents · 8.4 Summary

What you learn in chapter 8 →

Chapter 9: Creating an LLM project: Reimplementing Llama 3

The chapter introduces a hands-on project that brings earlier LLM lessons together by implementing and training a Llama-inspired model, improving it, and deploying it.

9.1 Implementing Meta’s Llama · 9.1.1 Tokenization and configuration · 9.1.2 Dataset, data loading, evaluation, and generation · 9.1.3 Network architecture · 9.2 Simple Llama · 9.3 Making it better · 9.3.1 Quantization · 9.3.2 LoRA · 9.3.3 Fully sharded data parallel–quantized LoRA · 9.4 Deploy to a Hugging Face Hub Space · 9.5 Summary

What you learn in chapter 9 →

Chapter 10: Creating a coding copilot project: Integrating an LLM service into VS Code with RAG

The chapter introduces a project to build a Python-focused coding copilot in VS Code using a pretrained coding model, an LLM service, and local retrieval-augmented generation.

10.1 Our model · 10.2 Data is king · 10.2.1 Our VectorDB · 10.2.2 Our dataset · 10.2.3 Using RAG · 10.3 Build the VS Code extension · 10.4 Lessons learned and next steps · 10.5 Summary

What you learn in chapter 10 →

Chapter 11: Deploying an LLM on a Raspberry Pi: How low can you go?

The chapter introduces a hands-on project to run a locally accessible LLM service on a Raspberry Pi, using its limitations to teach practical deployment lessons.

11.1 Setting up your Raspberry Pi · 11.1.1 Pi Imager · 11.1.2 Connecting to Pi · 11.1.3 Software installations and updates · 11.2 Preparing the model · 11.3 Serving the model · 11.4 Improvements · 11.4.1 Using a better interface · 11.4.2 Changing quantization · 11.4.3 Adding multimodality · 11.4.4 Serving the model on Google Colab · 11.5 Summary

What you learn in chapter 11 →

Chapter 12: Production, an ever-changing landscape: Things are just getting started

The chapter introduction explains why the book emphasizes durable LLM production concepts and previews a review of takeaways, future directions, and closing thoughts.

12.1 A thousand-foot view · 12.2 The future of LLMs · 12.2.1 Government and regulation · 12.2.2 LLMs are getting bigger · 12.2.3 Multimodal spaces · 12.2.4 Datasets · 12.2.5 Solving hallucination · 12.2.6 New hardware · 12.2.7 Agents will become useful · 12.3 Final thoughts · 12.4 Summary

What you learn in chapter 12 →

Concepts explained in this book

language modelA large language model is an artificial intelligence model trained on large text collections to predict and generate human language.
fine-tuningFine-tuning is the process of continuing model training on task-specific data to adapt a pretrained model to a particular behavior or domain.
quantizationQuantization is the conversion of model parameters or computations from higher-precision numerical representations to lower-precision representations.
embeddingA vector is an ordered numerical representation of values in a mathematical space.
low-rank adaptationLow-rank adaptation is a parameter-efficient fine-tuning method that adapts a pretrained model through trainable low-rank updates while leaving most original parameters fixed.
prompt engineeringPrompt engineering is the practice of designing and refining input instructions to influence an AI model's behavior and output.
tokenizationTokenization is the process of dividing text into discrete units that can be represented and processed by a language model.
retrieval-augmented generationRetrieval-augmented generation is a technique that supplies a generative model with information retrieved from external sources at response time.
semanticsSemantics is the study of meaning in language, including the meanings of expressions and their combinations.
pragmaticsPragmatic context is contextual information about real-world tasks, expectations, and circumstances that affects how language is interpreted or generated.
tokenizerA tokenizer is a text-processing component that divides input text into discrete tokens according to specified rules.
datasetA dataset is an organized collection of data used for analysis, evaluation, or machine-learning development.
syntaxSyntax is the system of rules governing how words and other linguistic units are structured into phrases and sentences.
open-source modelA machine-learning model whose weights, code, or other defining artifacts are made available under terms permitting specified use, modification, or redistribution.
prompt injectionPrompt injection is an attack in which crafted input attempts to manipulate a language model into ignoring intended instructions or revealing unintended information.
compressionCompression is the reduction of a model's size and computational requirements while retaining acceptable behavior and accuracy.
distillationDistillation is a model-training technique in which a student model learns to reproduce behavior or outputs from a teacher model.
parameter-efficient fine-tuningParameter-efficient fine-tuning is a fine-tuning approach that adapts a model by training only a small subset of parameters or added parameters.
self-supervised learningSelf-supervised learning is a machine-learning approach that derives training targets from the input data itself rather than requiring manually supplied labels.
biasA systematic pattern in model outputs that reflects skewed data or assumptions and produces inaccurate or unfair representations.
reinforcement learning with human feedbackReinforcement learning from human feedback is a model-training approach that optimizes behavior using preference or evaluation signals supplied by people.
mixture of expertsA model composed of multiple specialized expert subnetworks whose participation is selected or weighted for each input.
transfer learningTransfer learning is a machine-learning technique that adapts knowledge learned from one task or dataset to another task or domain.
context windowContext length is the maximum amount of tokenized input and generated context that a language model can process in one request.
edge computingA computing architecture that processes data near its source or point of use to reduce dependence on remote services.
adversarial inputThe study and use of inputs or models designed to manipulate, evade, or expose weaknesses in machine-learning systems.
low-rank approximationA method that represents a matrix or parameter structure with a lower-rank approximation to reduce storage or computation.
pruningStructured pruning removes whole structural components of a neural network, such as filters, channels, or layers.
contractAn API is a defined interface through which software applications request capabilities or data from another software system.
AI agentA software system that uses artificial intelligence to perceive information, make decisions, and perform actions toward goals.
attentionAn attention mechanism dynamically weights relationships between elements of an input sequence according to their relevance to a current computation.
training from scratchTraining from scratch is the process of creating a model by learning all of its parameters from randomly initialized weights rather than adapting a pretrained model.
morphologyMorphology is the study of word structure and the ways words are formed from smaller meaningful units.
phoneticsPhonetics is the study of speech sounds, including their physical production, transmission, and perception.
streamingStreaming is the incremental retrieval and processing of a dataset without loading the entire dataset into memory.
vector databaseA vector database is a database optimized for storing and retrieving numerical vector representations using similarity-based search.
edge deviceAn edge device is a computing device located near the source of data or the user rather than in a centralized cloud or data center.
natural languageA human communication system expressed through speech or writing and governed by linguistic conventions.
Raspberry PiRaspberry Pi is a family of small, low-cost single-board computers, including the Raspberry Pi 5 model.
build-versus-buy decisionA decision framework for choosing between developing a capability internally and acquiring it from an external provider.
entailmentEntailment is information or an implication supported by context beyond the literal text that is explicitly stated.
morphemeA morpheme is the smallest unit of language that carries meaning or serves a grammatical function.
speech actA speech act is an utterance considered in terms of the communicative action it performs, such as requesting, asserting, or instructing.
few-shot learningFew-shot learning is the adaptation or prediction of a model from only a small number of labeled examples.
foundational modelA foundation model is a broadly pretrained model that can be adapted to multiple downstream tasks or domains.
instruct schemaAn instruct schema is a structured data format that separates an instruction, its relevant input or metadata, and the desired response.
data cleaningData cleaning is the process of detecting, correcting, removing, or managing inaccurate, inconsistent, incomplete, or unsuitable data.
transformer architectureA transformer architecture is a neural-network architecture that models sequence relationships primarily through attention rather than recurrence or convolution.
BERTA transformer-based language representation model that uses bidirectional context to learn text embeddings.
SlovenBERTcinaA language model trained on Slovak-language data for strong performance on selected language tasks.
privacyThe protection of data from unauthorized access, disclosure, misuse, or loss while respecting requirements governing personal information.
SSHSSH is a cryptographic network protocol for secure remote login and command execution over an untrusted network.
hallucinationA hallucination is generated content that is presented as plausible but is unsupported, inaccurate, or fabricated relative to the relevant facts or source material.
GPU metricA GPU metric is an operational measurement describing graphics-processor utilization, capacity, or behavior.
scalabilityAutoscaling is the automatic adjustment of computing resources or service instances in response to workload or capacity conditions.
feature storeA feature store is a system or abstraction for managing, documenting, versioning, governing, and serving machine-learning features for training and production.
ambiguitySemantic ambiguity is the condition in which an expression supports multiple possible meanings or interpretations.
data annotationData annotation is the process of attaching labels or metadata to data instances, often with awareness of their positions or spans.
probability distributionA data distribution is the pattern of frequencies or probabilities with which data values or feature combinations occur.
schemaA schema is a formal specification of the structure, fields, types, and constraints of data.
DCGMData Center GPU Manager is NVIDIA software for managing, monitoring, and collecting telemetry from data-center GPUs.
KEDAKEDA is an open-source Kubernetes project that scales workloads according to event sources and custom or external metrics.
PrometheusPrometheus is an open-source monitoring system that collects time-series metrics and supports querying and alerting.
toolA tool is an external software capability or service that a language model can invoke to perform operations or access information.
similarity searchSemantic search retrieves information according to the meaning represented by queries and documents rather than relying only on exact lexical matches.
throughputThroughput is the rate at which a system completes requests or produces outputs over a period of time.
promptA persistent prompt instruction applied to requests to establish consistent model behavior or interaction style.
ScaledObjectA ScaledObject is a KEDA custom resource that declares scaling behavior and triggers for a Kubernetes workload.
custom metrics APIThe custom metrics API is a Kubernetes API for exposing application-defined metrics associated with Kubernetes resources.
flappingFlapping is repeated oscillation between scaling states, such as repeatedly adding and removing service replicas.
HorizontalPodAutoscalerA horizontal pod autoscaler is a Kubernetes controller that adjusts an application's pod replica count according to observed metrics.
Prometheus AdapterPrometheus Adapter is a Kubernetes metrics adapter that exposes Prometheus metrics through Kubernetes metrics APIs.
LlamaLLaMA is a family of autoregressive large language models released by Meta for research and development.
positional embeddingPositional encoding is a method of adding sequence-position information to token representations in a transformer model.
chain-of-thought promptingChain-of-thought prompting encourages a language model to produce or use intermediate reasoning steps before giving a final answer.
singular value decompositionSingular value decomposition is a matrix factorization that expresses a matrix as a product of orthogonal and diagonal matrices.
data parallelData parallelism is a parallel-computing technique that processes separate portions of data concurrently with multiple copies of a model or pipeline.
n-dimensional parallelism3D parallelism is a distributed-training strategy that combines data parallelism, tensor parallelism, and pipeline parallelism.
pipeline parallelismPipeline parallelism is a distributed-computing technique that partitions a model into sequential stages placed on different devices.
tensor parallelismTensor parallelism is a distributed-computing technique that partitions tensor computations across multiple processors and combines their results.
code compilationCompilation is the transformation of high-level program or model code into lower-level code optimized for execution on target hardware.
intermediate representationAn intermediate representation is a machine-readable form of a program or model that enables translation and optimization between source frameworks and target hardware.
GPU kernelA kernel is a small program or function executed on a parallel processor to perform a computational operation.
kernel tuningKernel tuning is the selection and optimization of processor kernels to minimize execution time for a specific workload and hardware configuration.
ONNXONNX is an open model format and ecosystem for representing neural-network models across different frameworks, tools, and hardware environments.
ONNX RuntimeONNX Runtime is a cross-platform inference engine that optimizes and executes models represented in ONNX format.
trajectoryModel tracing is the capture of operations invoked during a model execution to construct an intermediate computational representation.
tensor fusionTensor fusion is the merging of multiple tensor-operation kernels into a combined operation to reduce launches and improve memory efficiency.
TensorRTTensorRT is NVIDIA software for optimizing and executing machine-learning inference on NVIDIA GPUs.
training loopA training loop is the repeated process of computing model outputs, evaluating loss, calculating gradients, and updating parameters.
out-of-vocabulary wordAn unknown token is a reserved token representing a word or text unit that a tokenizer or model does not recognize.
input spaceThe input space is the range and maximum amount of input content that a model can accept for a single computation.
language modelingLanguage modeling is the task of learning the statistical structure of language so a system can predict or generate text.
batch sizeBatch size is the number of input examples or inference requests processed together in one computation batch.
emergent behaviorEmergent behavior is a capability or pattern that appears in a model at sufficient scale despite not being observed at smaller scales.
DSPyDSPy is a framework for programming and optimizing language-model prompts and pipelines through declarative modules and evaluation-driven optimization.
guardrailA guardrail is a constraint or control intended to keep a language model's behavior within acceptable safety or policy boundaries.
multi-gpu environmentA multi-GPU environment is a computing setup that provides access to multiple graphics processing units for distributed or parallel machine-learning workloads.
one-shot promptingOne-shot prompting is a prompting technique that supplies exactly one example or instruction to guide a language model's performance on a task.
GuidanceGuidance is an open-source library from Microsoft for enforcing programmatic constraints on language-model responses.
AI-generated content detectionAn AI detection system is software intended to identify content or activity generated or assisted by artificial intelligence.
copyrightA legal right that grants creators control over the reproduction and distribution of their original works.
copyright infringementUnauthorized use, reproduction, or distribution of a copyrighted work.
ethicsPrinciples concerning morally acceptable behavior and the consequences of technological decisions.
intellectual propertyIntellectual property is a legally protectable intangible creation, information asset, or other proprietary business resource.
liabilityLegal responsibility for harm, loss, or an obligation resulting from an action or omission.
regulationA regulation is a rule or legal requirement established by a government or other authority to govern conduct or activities.
model architectureA model architecture is the structural design of a machine-learning model, including its computational components and their connections.
VS Code extensionA VS Code extension is a packaged software component that adds capabilities to the Visual Studio Code editor.
Visual Studio CodeVS Code is a source-code editor from Microsoft with extensibility for programming, debugging, and remote development.
benchmarkA benchmark dataset is a curated dataset used to evaluate and compare model performance on defined tasks.
productionAn operational environment in which a software system or machine-learning model is live and available to perform real work.
distributed systemDistributed computing is a computing technique that divides workloads across multiple networked devices or nodes for concurrent execution.
zero-shot learningA learning or inference setting in which a model performs a task without being given task-specific examples.
RedPajamaRedPajama is a large open text dataset designed to reproduce the data composition described for the LLaMA model.
GPT modelA GPT model is a generative pretrained transformer language model that produces text from an input context.
llm serviceAn LLM service is a deployed software service that exposes a large language model for remote requests and response generation.
LangChainLangChain is a software framework for building applications that combine language models with prompts, tools, data sources, and workflow logic.
abstractionAn abstraction is a generalized representation that captures relevant properties while omitting particular details.
batchingA microbatch is a subdivision of a larger batch that is processed as one unit within a parallel training pipeline.
idle percentageIdle percentage is the proportion of pipeline execution time during which processing devices are not performing useful computation.
RayRay is an open-source framework for executing distributed and parallel Python applications across clusters.
sequence parallelismSequence parallelism is a distributed-computing technique that partitions neural-network activations along the sequence dimension to reduce redundant storage.
training datasetTraining data is the collection of examples from which a machine-learning model learns its parameters or behavior.
monitoringA monitoring system observes a machine-learning system's data, resources, behavior, and outputs to detect degradation, failures, and operational issues.
subword tokenizationA subword is a tokenization unit corresponding to part of a word rather than necessarily a complete word.
experiment trackingAn experiment tracker is a tool that records machine-learning experiments, configurations, artifacts, and results so they can be inspected and compared.
trainingTraining is the process of optimizing a machine-learning model's parameters from data so it learns a desired behavior or capability.
vectorizationVectorization is the transformation of input data into numerical vector representations.
JAMBAJAMBA is a hybrid language-model architecture combining state-space and transformer layers with joint attention.
MambaMamba is a sequence-modeling architecture based on selective state-space mechanisms and designed to avoid standard attention costs.
speculative decodingSpeculative decoding accelerates generation by having a smaller assistant model propose tokens that a larger model verifies in parallel.
conversational aiAn AI assistant is software that interacts with users and performs tasks through artificial-intelligence capabilities.
groundingGrounding is the practice of supplying a language model with relevant external context so its generation is anchored to available information.
knowledge editingKnowledge editing is the technique of modifying specific factual or behavioral associations in a trained model without retraining the entire model.
knowledge graphA knowledge graph represents entities and the relationships between them as a graph of nodes and edges.
coding copilotA coding copilot is a software assistant that generates or suggests source code in response to a developer's context or request.
extension manifestAn extension manifest is a configuration document that declares an editor extension's identity, capabilities, activation behavior, and contributions.
ghost textGhost text is provisional text rendered inline in an editor as a preview of a suggested completion before the user accepts it.
inline completion itemAn inline completion item is a proposed text completion that a VS Code inline suggestion provider returns for display in the editor.
inline suggestion providerAn inline suggestion provider is a VS Code extension component that supplies completion suggestions displayed at the cursor position.
package.jsonpackage.json is a JSON configuration file that declares metadata, dependencies, scripts, and other package information for a JavaScript project.
VS Code APIThe VS Code API is the programming interface through which extensions interact with Visual Studio Code and its editor features.
VS Code extension template generatorA VS Code extension template generator is a tool that creates starter project files and scaffolding for a Visual Studio Code extension.
natural language interfaceChatInterface is a Gradio component for creating a conversational user interface around a function.
data leakageData leakage is the inclusion of evaluation or otherwise unavailable information in a model's training data or training process.
DeciCoderDeciCoder is a language model designed to generate source code, including Python, JavaScript, and Java code.
ToolFormerToolformer is a language-model training approach in which models learn to decide when and how to call external tools.
logitA logit is an unnormalized real-valued score assigned to a candidate token before probability normalization.
argmaxArgmax is the operation that returns the position of the maximum value in a set or vector.
container imageA Docker image is an immutable package containing a filesystem and metadata used to create Docker containers.
Google Cloud SDKThe gcloud CLI is Google's command-line interface for managing Google Cloud resources and services.
Google Compute EngineGoogle Compute Engine is Google Cloud's infrastructure service for creating and running virtual machines.
Remote-SSHRemote-SSH is a VS Code extension that provides an editor interface for working with files and processes on SSH-accessible remote hosts.
SCPThe Secure Copy Protocol is a protocol for securely copying files between hosts over an SSH connection.
virtual machineA virtual machine is a software-defined computer that runs an operating system and applications in an isolated virtualized environment.
vocabularyA vocabulary is the set of tokens and their associated identifiers recognized by a tokenizer or language model.
zero-shot promptingZero-shot prompting is prompting a model to perform a task without providing task-specific examples in the prompt.
normalizationNormalization is the transformation of data values to a common scale or standardized representation.
GGUFGGUF is a model-file format designed to store quantized language models for efficient local loading and inference.
ChatGPTChatGPT is a conversational AI service that provides an interactive interface for generating responses from large language models.
byte pair encodingByte pair encoding is a subword tokenization algorithm that iteratively merges frequent adjacent symbol pairs into larger vocabulary units.
SentencePieceSentencePiece is a library and tokenization framework that learns subword vocabularies directly from raw text without requiring pretokenization by whitespace.
mlopsMachine learning operations is the discipline of reliably deploying, monitoring, maintaining, and governing machine-learning systems throughout their life cycle.
multimodalA multimodal solution is an AI system or method that processes or combines information from multiple data modalities.
CUDA toolkitThe CUDA Toolkit is NVIDIA's software development environment for developing and running GPU-accelerated applications using CUDA.
ReLUA rectified linear unit is an activation function that outputs zero for negative inputs and otherwise passes through the input.
encoderAn encoder is the context-processing component of a transformer architecture that converts input tokens into contextual representations.
top-p filteringTop-p sampling restricts token selection to the smallest candidate set whose cumulative probability reaches a specified threshold.
pretrainingPretraining is the initial training of a model on a large, broad dataset before adapting it to a specific task or domain.
contextInformation supplied with a prompt that enables a language model to interpret a request and produce an appropriate response.
instructionThe instruction component of a prompt is the task-specific directive that specifies what the language model should do.
n-gramAn n-gram is a contiguous sequence of n linguistic units, commonly used to model local patterns in text.
deployment serviceA deployment service is a software service that configures, provisions, and exposes trained machine-learning models for inference in an operational environment.
rate limitA rate limiter is a mechanism that restricts how many requests a client can make to a service within a specified time interval.
gatingA gating mechanism selectively controls how information enters, remains in, or leaves a recurrent model state.
minimum token frequencyMinimum token frequency is the threshold number of dataset occurrences required for a token to enter a tokenizer vocabulary.
special tokenA special token is a reserved token with a syntactic or control function in tokenized model input.
degradationPerformance decay is a decline in a model's output quality over time despite the model continuing to operate.
model registryA model registry is a system for storing, versioning, annotating, and controlling access to trained machine-learning models.
learning rate scheduleLearning rate scheduling is the practice of changing an optimizer's learning rate during training according to a predefined or adaptive schedule.
objective functionA loss function is an objective that quantifies discrepancies between a model's predictions and target outputs for optimization during training.
OpenAPI specificationOpenAPI is a specification standard for describing HTTP APIs so that their operations, inputs, outputs, and security requirements can be understood by tools and clients.
pluginOpenAI Plugins are extensions that let ChatGPT access third-party tools and external APIs.
feedforward networkA feed-forward layer is a neural-network layer that transforms its input through learned operations without recurrent state.
TinyStoriesTinyStories is a small text dataset of short synthetic stories intended for training and experimenting with language models.
multinomialMultinomial sampling selects a categorical outcome randomly according to its probability distribution.
validation lossValidation loss is the loss computed on held-out validation examples to assess a model during development.
structured memory storageA technique that organizes retained information so an agent can retrieve the most relevant details for subsequent steps.
LSTMA long short-term memory network is a recurrent neural network architecture that uses memory cells and gates to preserve or discard information across sequences.
recurrent neural networkA recurrent neural network is a neural network that processes ordered inputs while maintaining a state that carries information across sequence positions.
context-aware filteringContext-aware filtering evaluates input or output in its surrounding semantic context to detect unsafe, unauthorized, or policy-violating content.
poisoning attackData poisoning is an attack that inserts malicious or manipulated examples into training data to alter a model's behavior.
securitySecurity is the protection of systems, data, and operations against unauthorized access, misuse, disruption, or compromise.
semioticsSemiotics is the study of signs, symbols, and the processes by which they convey and are interpreted as meaning.
llama.cppllama.cpp is an open-source software project for running and deploying large language models locally, especially on commodity hardware.
FastAPIFastAPI is a Python web framework for building HTTP APIs with type-based request validation and automatic interface documentation.
parameterA model parameter is a learned numerical value that determines part of a machine-learning model's computation.
GLUE BenchmarkGLUE is a standardized benchmark suite for evaluating language-model performance on diverse language-understanding tasks.
MMLUMMLU is a benchmark suite that evaluates broad language understanding, world knowledge, and problem-solving across subjects and difficulty levels.
SuperGLUESuperGLUE is a benchmark suite with more difficult and diverse language-understanding tasks than GLUE.
multimodal projection modelA multimodal projection model converts representations from a non-language modality into a form that a language model can process.
BLEU scoreBLEU is an evaluation metric for machine translation that measures modified n-gram precision against one or more reference translations.
boot timeBoot time is the elapsed time required for a machine or service instance to start and become ready for operation.
BPCBPC is an entropy-based language-model evaluation metric measuring the average number of binary digits needed to represent each character.
baking modelBaking a model is the technique of packaging model files inside a container image so newly created containers contain the model.
fusingFusing is the technique of mounting remote object storage on a machine so it appears and behaves like a local filesystem.
intermediary mounted volumeAn intermediary mounted volume is a storage strategy in which a model is downloaded once and retained on a volume that later instances can mount.
mounted volumeA mounted volume is a storage resource attached to a running machine or container and exposed through its filesystem.
object storeAn object storage system stores data as independently addressable objects distributed across storage infrastructure.
precisionPrecision is the proportion of predicted items that are correct among all predicted items.
recallRecall is the proportion of reference items that are correctly identified by a prediction.
ROUGEROUGE is a family of automatic evaluation metrics that measure lexical overlap between generated text and reference text, commonly for summarization.
dialogueChat history is the ordered record of messages exchanged during a conversational interaction.
EvaluateEvaluate is a Hugging Face library for building and applying machine-learning evaluation methods and metrics.
ReActReAct is a prompting framework that interleaves a language model's reasoning with external actions and observations to solve tasks.
driftData drift is a change over time in the statistical properties or distribution of data presented to a machine-learning system.
DataOpsDataOps is a discipline for managing the development, movement, quality, and operation of data systems and pipelines.
paddingPadding is the addition of placeholder elements to sequences so that inputs in a batch share a common length.
feedbackHuman feedback is an evaluation, preference, or correction supplied by people about a system's behavior or output.
continuous modelContinuous language modeling predicts target words from distributed representations of surrounding context rather than only discrete word counts.
load testingLoad testing is a performance-testing technique that evaluates how a service or system behaves under an expected or increasing workload.
LocustLocust is an open-source load-testing tool for defining and executing distributed user-behavior simulations.
replicaA replica is an additional running instance of a service or workload that provides capacity or redundancy.
time to first tokenTime to first token is the elapsed time between submitting a language-model request and receiving its first generated token.
bidirectional rnnBidirectionality is the processing of a sequence in both forward and reverse directions and combining the resulting representations.
dropoutDropout is a regularization technique that randomly disables neural-network units during training to reduce co-adaptation and overfitting.
exploding gradientAn exploding gradient is a training condition in which propagated derivatives grow excessively large and destabilize optimization.
memory cellA memory cell is a persistent state component in a long short-term memory network that stores information across sequence positions.
sequenceSequence data is data whose elements have an ordered position that carries information.
vanishing gradient problemA vanishing gradient is a training condition in which propagated derivatives become increasingly small and impede parameter updates.
decodeA decoder is a neural-network component that generates an output sequence from an internal representation and, typically, previously generated tokens.
token limitA token limit is a maximum number of tokens that a language-model API or computation accepts or processes for a request.
meaning-makingMeaning-making is the process by which people or systems construct, assign, and communicate significance.
n-gram modelAn n-gram model is a probabilistic language model that estimates word sequences using a fixed-length context of n tokens.
templateA reusable prompt structure containing instructions and variable content for directing a language model.
semiotic triangleA semiotic triangle is a conceptual model that organizes relationships among signs, their forms, and what they represent or mean.
signA sign is something that conveys meaning by standing for or indicating something else.
signifierA signifier is the perceptible form or representation through which a sign conveys meaning.
autoscalingScaling down is the deliberate reduction of active service capacity or computing resources.
AccelerateAccelerate is a Hugging Face library that simplifies running machine-learning code across multiple devices and distributed environments.
bitsandbytesbitsandbytes is a library providing low-bit optimizers and quantization operations for reducing the memory requirements of neural-network models.
DeepSpeedDeepSpeed is an optimization library for distributed deep learning that provides training and inference efficiency features.
memory-efficient attention mechanismMemory-efficient exact attention is an attention computation method that reduces memory use while preserving exact attention results.
xFormersxFormers is a modular library of optimized and experimental components for transformer-based deep-learning models.
ChainlitChainlit is a software tool for building applications with large language models and conversational interfaces.
fallbackA fallback response is a predefined response returned when processing a user request fails.
feedback formA feedback form is a user-interface component that collects structured or free-form responses from users.
GradioGradio is an open-source Python library for building interactive web interfaces around machine-learning functions and models.
LLaVALLaVA is a multimodal large language model that combines language understanding with visual input processing.
backpropagationBackpropagation is the procedure for computing parameter gradients by propagating error derivatives backward through a neural network.
character-based modelingCharacter-based modeling is language modeling in which text is represented and predicted as sequences of individual characters.
epochAn epoch is one complete pass through a machine-learning training dataset.
overfittingOverfitting is a condition in which a model becomes excessively tailored to training data and performs poorly on unseen data.
tokenA token is a discrete unit of text representation processed by a language model.
evaluation datasetA validation dataset is a held-out collection of examples used to assess and tune a model during development.
cost engineeringCost engineering is the systematic planning, estimation, monitoring, and control of costs across a system's lifecycle.
inference graphAn inference graph is a graph-based deployment pattern that connects models and services into a coordinated inference workflow.
SeldonSeldon is an open-source platform for deploying and managing machine-learning models, including in Kubernetes-based environments.
LLMOpsLLMOps is the discipline of developing, deploying, monitoring, and maintaining applications and systems built around large language models.
hyperparameter tuningHyperparameter tuning is the process of selecting values for training or model-configuration choices that are not learned as model parameters.
infrastructureInfrastructure is the hardware, software, networking, and operational resources that support computing systems.
gpu shortageA GPU shortage is a condition in which demand for available graphics processors exceeds their supply or accessibility.
learning rateA learning rate is a training hyperparameter that controls the magnitude of parameter updates during optimization.
reward signalA reward system is a mechanism that assigns feedback signals to actions or outcomes to guide an agent's behavior.
cachingCaching is the technique of storing reusable computation results or data so later requests can be served without repeating the original work.
gRPCgRPC is an open-source remote procedure call framework and protocol for invoking operations across process or network boundaries.
GPUA GPU is a processor designed for highly parallel numerical computation, commonly used to accelerate machine-learning workloads.
natural language processingNatural language processing is the field concerned with computationally processing and understanding human language.
versioningVersioning is the practice of recording and managing distinct revisions of software, data, models, or configuration.
naive bayes classifierA naive Bayes classifier is a probabilistic classification model that applies Bayes' theorem while assuming conditional independence among features given the class.
SwiGLUSWIGLU is a gated activation function that uses the Swish function to modulate one linear projection with another.
stateA hidden state is an internal representation produced by a sequence model that summarizes information processed up to a position or layer.
Kubernetes sidecarA Kubernetes sidecar is a companion container running in the same pod as an application container to provide supporting functionality.
sidecar processA sidecar is a companion component deployed alongside a primary application to perform supporting processing or communication.
DevOpsDevOps is the set of practices that integrates software development and IT operations to automate and improve software delivery and operation.
machine learning workflowA machine learning life cycle is the sequence of activities involved in creating, deploying, operating, and retiring machine-learning models.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.