What you will learn in LLM Customization and Fine-Tuning
Adaptation, distillation, and alignment
Chapter by chapter
Chapter 1: Why model adaptation?
The introduction explains why general-purpose LLMs can fail at enterprise tasks and defines model adaptation as a range of ways to close that gap.
What you learn in chapter 1 →Chapter 2: How to do model adaptation
The chapter introduction uses an earnings-transcript failure to show why adaptation methods must be chosen for the task rather than by how intensive they are.
What you learn in chapter 2 →Chapter 3: What data do I need for model adaptation?
This chapter introduction explains why having enterprise data is not the same as having data suitable for model adaptation, using IT support records to show why quality and representativeness matter.
What you learn in chapter 3 →Chapter 4: In-context learning, few-shot, and RAG
The chapter opens by using a ticket-routing example to explain in-context learning and why a few-shot baseline is worth testing before fine-tuning.
What you learn in chapter 4 →Chapter 5: Parameter-Efficient Fine-Tuning: LoRA and QLoRA
Introduces PEFT as a lower-cost way to adapt a frozen model, explains LoRA’s adapter-based approach, and identifies when PEFT is useful.
What you learn in chapter 5 →Chapter 6: Supervised fine-tuning: Maximum expressiveness
Introduces full supervised fine-tuning as a more expressive but costlier option when a LoRA adapter falls short of a measured quality target.
What you learn in chapter 6 →Chapter 7: Knowledge distillation: Capturing frontier model intelligence
Introduces knowledge distillation as training a smaller student on a stronger teacher’s outputs, explains its enterprise appeal, and flags the need to preserve specialized behavior.
What you learn in chapter 7 →Chapter 8: Preference optimization: Teaching your model to judge
This introduction explains why preference optimization follows supervised fine-tuning when a model needs to distinguish between acceptable responses of different quality.
What you learn in chapter 8 →Chapter 9: Managing model evolution, drift, and versioning
This introduction explains why a deployed IT support model needs ongoing monitoring and maintenance even when its own weights have not changed.
What you learn in chapter 9 →Concepts explained in this book
- supervised fine-tuningFurther training a pretrained model on labeled input-output examples to adapt its behavior.
- low-rank adaptationA learned, low-rank additive change to a model's weights.
- distillationTraining a student model to imitate the behavior of a teacher model, often a larger or stronger one.
- prompt engineeringSteering a model's behavior through its input instructions or examples without updating its weights.
- retrieval-augmented generationA technique that retrieves relevant external information and supplies it to a generative model when producing a response.
- token-F1An F1 score based on token-level precision and recall between a predicted text and a reference text.
- evaluation datasetA subset of examples reserved for evaluating a model rather than updating its parameters.
- adapterA small trainable component added to a model to modify its behavior without updating all its original parameters.
- Direct Preference OptimizationA method that tunes a model directly from preference comparisons without separately training a reward model.
- preference tuningTraining a model from preference signals so that it favors desired responses over less desired ones.
- model adaptationModifying a pretrained model for a particular task, domain, or use case without training it from scratch.
- data qualityThe degree to which adaptation data is correct, relevant, and representative enough to support reliable model behavior.
- student modelA student model is a smaller model trained to mimic a teacher model during distillation.
- fine-tuningFurther training a pretrained model on task- or domain-specific data to change its behavior.
- preference pairA training example containing two responses to the same prompt and a label indicating which is preferred.
- SFTTrainerA trainer component in the TRL library for supervised fine-tuning of transformer language models.
- regressionA change that makes a model less likely to decline requests outside its competence.
- refusal rateRefusal rate is the proportion of tested requests that a model refuses, with the request set determining whether it measures appropriate or harmful-request refusals.
- demonstrationA data example consisting of an input prompt and its desired completion.
- validation lossA measure of a model’s training objective on held-out data, used to assess how well it generalizes beyond its training examples.
- capability retentionThe student model's score divided by the teacher model's score on the same evaluation metric and set.
- safety regression suiteThe practice of repeatedly evaluating model changes for newly introduced safety failures.
- adaptation continuumA range of model-adaptation options, from prompting and retrieval to fine-tuning, considered as choices with different costs and complexity.
- teacher modelA model whose outputs or behavior provide targets for training a student model during distillation.
- in-context learningA technique in which a model uses examples and instructions in its prompt to perform a task without updating its weights.
- QLoRAA LoRA-based fine-tuning method that loads the base model in low-bit precision to reduce memory use during adaptation.
- LoRA rankThe rank hyperparameter that sets the dimensionality of a LoRA adapter’s low-rank updates.
- training datasetThe collection of examples used to fit or adapt a model during training.
- synthetic dataArtificially generated examples used for training or evaluation.
- objective functionA measure of how well a model’s predictions fit its training examples, typically computed from the training objective.
- baselineA reference performance level against which model changes are evaluated.
- LLM-as-a-judgeAn evaluation setup in which a language model judges other model outputs against specified criteria.
- test suiteAn organized collection of test cases designed to evaluate a system against defined criteria.
- alignment trainingSteering a model's behavior toward human or organizational preferences and constraints.
- many-shot promptingPrompting with a relatively large number of examples to demonstrate the desired behavior.
- parameter-efficient fine-tuningAdapting a pretrained model by updating only a small number of added or selected parameters rather than all its weights.
- privacyThe requirement that sensitive data not be exposed to external systems during model use or deployment.
- gradient checkpointingA memory-saving training technique that discards intermediate activations and recomputes them as needed during backpropagation.
- drift detectionA method or component that identifies changes in data or system behavior over time.
- rollbackReverting a deployed system to an earlier version when the current version causes problems.
- data driftA change over time in the characteristics or distribution of data a model encounters, potentially reducing its performance.
- Group Relative Policy OptimizationA reinforcement-learning method that optimizes a policy using relative comparisons among groups of sampled outputs instead of a separate critic network.
- quality gateA check that determines whether generated or collected examples meet quality requirements before they are accepted into a dataset.
- costThe money required to train, operate, or maintain a model.
- exact-match accuracyExact match is a metric that assigns a binary score based on whether a generated response exactly matches a reference after normalization.
- overfittingA model-learning failure in which performance worsens on inputs that differ from the training distribution because the model has fit the training data too narrowly.
- refusal behaviorA model's act of declining to fulfill a request, often because complying would be inappropriate or unsafe.
- learning rateThe learning rate is a training hyperparameter that controls the size of parameter updates during optimization.
- quality filterA screening process that removes low-quality examples before they are used for training.
- accuracyThe proportion of evaluated predictions or answers that are correct.
- embeddingA numerical vector representation of an item that encodes features useful for computational comparison.
- knowledge gapA limitation in which a model lacks information needed to answer a question.
- behavior gapA model limitation in which it knows the relevant fact but responds in an undesired format, tone, or level of detail.
- model registryA system for storing model artifacts and metadata such as versions, training details, and evaluation records for audit and deployment control.
- return on investmentReturn on investment: the economic benefit of an investment relative to its cost.
- self-hostingA deployment arrangement in which systems run within an organization's own infrastructure rather than an external provider's environment.
- driftA change over time in inputs, their relationship to correct outputs, or model behavior that can degrade performance.
- run-to-run noiseA measure of how often repeated model runs produce different predictions for the same input.
- weak supervisionA data-labeling approach that produces training labels from noisy, indirect, or programmatic sources rather than manual annotation.
- reinforcement learning with verifiable rewardsA method of fine-tuning a model with reinforcement learning using feedback on its outputs.
- latencyThe time between sending a request and receiving a response, including network and processing delays.
- reference modelA fixed policy distribution that serves as the comparison point for constraining changes to an optimized policy.
- provenanceThe record of data’s origins, transformations, and movement through processing systems.
- cosine similarityA measure of the angle-based similarity between two vectors, often used to compare weighted text representations.
- canary promptA fixed input submitted periodically to a model so its output can be compared with a saved baseline.
- quantizationRepresenting model values at reduced numerical precision to lower storage or computational requirements.
- monitoringThe ongoing observation of a deployed model to detect performance issues, failures, or drift in production.
- foundation modelThe original pretrained model whose weights are kept fixed while LoRA learns an additive adaptation.
- refusal exampleA refusal example is a training instance that demonstrates declining a harmful or disallowed request.
- schemaA fixed arrangement of fields and separators used to present examples and queries in a consistent format for a model.
- Hit@1The proportion of queries for which at least one relevant result appears among the top k retrieved results.
- catastrophic forgettingThe loss of previously learned capabilities when a model is fine-tuned on narrower data, as updates overwrite parameters supporting older skills.
- managed APIA provider-operated model interface accessed through an API rather than self-hosted by the user.
- text chunkA segment of a document that can be indexed or returned as an individual retrieval result.
- preference dataA collection of judgments or comparisons expressing preferences between candidate model outputs.
- versioningThe practice of tracking distinct iterations of a model and their changes over time.
- hyperparameterA configurable setting that controls model training rather than being learned directly from training data.
- personal dataPersonal data is information relating to an identified or identifiable individual.
- preference marginThe difference between the implicit rewards assigned to a chosen response and a rejected response in a preference pair.
- reward accuracyThe proportion of preference pairs for which a model assigns a higher implicit reward to the chosen response than to the rejected response.
- LoRA scaling factorA LoRA hyperparameter that scales the contribution of the low-rank adapter to the model.
- memorizationA training outcome in which a model learns training examples without correspondingly improving its ability to generalize to new examples.
- inter-rater reliabilityThe degree to which different annotators assign the same labels or answers to the same examples.
- reasoning modelA model optimized to perform multi-step reasoning tasks, especially those requiring structured deliberation.
- data sovereigntyThe control of sensitive data within an organization's own infrastructure or jurisdiction so it does not leave the trusted boundary.
- gradient normThe magnitude of a model’s gradient vector, which summarizes the size of parameter-update signals during training.
- spot-checkingHuman review of a sample of examples to identify quality problems that automated checks may miss.
- data acquisition strategyThe plan for collecting, selecting, and preparing data to support model adaptation.
- reward functionA function that assigns a numerical score to an output, action, or sequence of actions to represent its quality.
- data annotationThe process of adding labels or other explanatory information to data so it can support model development or evaluation.
- dataset versioningThe practice of tracking and preserving distinct dataset versions for traceability and reproducibility.
- prompt cachingA technique that stores reusable prompt prefixes or inputs so repeated content does not need to be processed or billed in the same way each time.
- retrieval layerThe supporting systems required to perform retrieval for RAG, such as indexing and vector storage.
- chunkingThe process of dividing documents into smaller units for indexing or retrieval.
- KL regularizationA constraint based on Kullback–Leibler divergence that limits how far a policy distribution can move from a reference distribution.
- biasA systematic skew in model behavior or outcomes.
- fairnessThe quality of treating people or groups equitably in a model’s data, behavior, and outcomes.
- reinforcement learningA machine-learning approach in which an agent learns a policy through reward signals from interactions or sampled outcomes.
- vector indexAn index structure that stores vectors and enables fast nearest-neighbor or similarity search.
- DPOTrainerA TRL trainer for optimizing transformer language models from preference pairs using Direct Preference Optimization.
- sentence-transformer embeddingA transformer model that maps sentences to vector representations designed to capture semantic similarity.
- failure modeA recurring way in which a retrieval-augmented generation system can produce incorrect or degraded behavior in production.
- hybrid searchA retrieval approach that combines multiple methods, commonly lexical search and vector similarity search.
- rerankingA retrieval technique that reorders an initial set of results using a further relevance assessment.
- concept driftA change in the relationship between a model’s input data and the concepts or outcomes it is expected to represent.
- self-distillation fine-tuningA fine-tuning method in which a model conditioned on an expert demonstration trains the same model conditioned on the prompt alone.
- disagreementThe proportion of cases where model outputs diverge across runs, prompts, or comparison conditions, used here as a brittleness indicator.
- full fine-tuningFine-tuning that updates the model's full set of weights rather than only a subset or adapter.
- break-even pointThe point at which two alternatives have equal total cost under the given assumptions.
- chosen responseThe candidate response in a preference pair that a human or automated system prefers.
- rejected responseThe candidate response in a preference pair that is less preferred than its paired alternative.
- instruct modelA language model fine-tuned to reliably follow instructions or prompts.
- automated filterThe use of programmatic checks to identify and remove examples that fail specified quality criteria.
- dataset splitA subset of a dataset assigned a role such as training, validation, or testing.
- data residencyA requirement that data be stored or processed within specified geographic or organizational boundaries.
- verificationThe process of checking generated training data for correctness, usefulness, and suitability before training.
- promptThe input supplied to a model, potentially containing instructions, context, and examples.
- approximate nearest neighbor searchAn indexing/search approach that trades a small amount of recall for much faster nearest-neighbor retrieval.
- corpusA body of text or other training data used to train a model or, here, to build the retrieval index for RAG.
- embedding-dimension driftThe incompatibility that arises when an embedding model changes and old vectors can no longer be compared directly with new ones.
- FAISSA vector search library used here as an in-process index implementation and as a backend for IVF.
- HNSWA graph-based approximate nearest-neighbor indexing method commonly used for fast vector search.
- PineconeA managed vector database platform used here as an example of HNSW-based production vector search.
- information retrievalThe process of finding the most relevant text chunks for a query based on embedding similarity.
- vector databaseAn operational database layer that provides vector search plus persistence, filtering, replication, multi-tenancy, and observability.
- operational independenceThe ability to operate a model without dependence on an external provider or its service availability, limits, or policies.
- Kullback–Leibler divergenceA measure of how one probability distribution differs from a reference probability distribution.
- reverse KL divergenceThe Kullback–Leibler divergence from a student distribution to a teacher distribution.
- drift scoreA numerical measure of the difference between reference and production text distributions, computed here from centroid cosine similarity.
- PII scrubbingThe removal or masking of personally identifiable information from data.
- smoke testA smoke test is a basic test that checks whether essential parts of a software system run successfully.
- qualitative assessmentAssessment of model outputs by examining their characteristics rather than relying only on numerical scores.
- AI-ready dataData prepared and curated so it is suitable for training or adapting a model, rather than merely existing in an operational system.
- fidelityA measure of whether a sentiment label matches the content of its sentence.
- adaptation techniqueThe specific method used to modify a base model, such as prompting, RAG, LoRA, fine-tuning, distillation, or alignment.
- anonymizationThe process of removing or transforming identifying information so data cannot readily be linked to a specific person.
- data curationThe process of selecting, cleaning, organizing, and preparing data for adaptation or evaluation.
- chain-of-thought promptingA prompting technique that elicits intermediate reasoning steps from a language model before its final answer.
- NormalFloat4A four-bit quantization format designed for neural-network weights that are approximately normally distributed.
- RLHFA training approach that uses human preference feedback to shape a model, typically through a reward model and reinforcement-learning optimization.
- clarificationA response that asks for missing information when a request is underspecified.
- canary checkA check of model behavior on selected prompts to detect changes or regressions in its responses.
- recallA measure of how much of the probability mass or set of outcomes represented by a teacher distribution is captured by a student distribution.
- edge caseAn uncommon or difficult input situation that tests whether a model behaves correctly beyond routine scenarios.
- reward modelA model trained to assign scores representing the desirability of outputs, often from human preference judgments.
- annotation provenanceInformation recording the human labeling history and review details associated with a data example.
- source provenanceInformation recording which source a data example came from and when it was collected or generated.
- transformation provenanceInformation recording the automated transformations applied to a data example.
- recency ruleA prompt-ordering heuristic that places the example most similar to the query last.
- brand voiceThe distinctive tone, style, and messaging an organization wants its text to convey.
- structured extractionThe process of identifying and extracting specified fields from unstructured or semi-structured sources.
- PEFT libraryA Hugging Face library for parameter-efficient adaptation of pretrained models.
- regression testA test that checks whether a change has caused previously acceptable behavior or performance to deteriorate.
- seed datasetA small set of initial examples used to condition or guide later synthetic generation.
- coverage gapA shortfall in the range of cases or situations represented by examples or a dataset.
- similarity searchRetrieval that selects items according to their similarity to a query or example.
- prompt templateA reusable specification that organizes instructions and other content into a model prompt.
- greedy decodingGreedy decoding is a generation method that selects the highest-probability next token at each step.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
