Chapter 6: Supervised fine-tuning: Maximum expressiveness
Introduces full supervised fine-tuning as a more expressive but costlier option when a LoRA adapter falls short of a measured quality target.
After this chapter you will be able to
- Full SFT versus LoRA updates. Explain how full SFT differs from LoRA in which model parameters it trains and why that gives SFT more freedom to reshape behavior.
- SFT escalation and costs. Justify when the section recommends trying full SFT instead of continuing with LoRA, and describe the associated training costs and risk.
- Full Model SFT Updates. Explain how this SFT workflow differs from LoRA: it omits the LoRA configuration and updates the full model rather than training an adapter, aiming to make support behavior, step-by-step formatting, and domain vocabulary the model’s defaults.
- Balanced training example mix. Explain why the pipeline mixes general-purpose examples into the IT data and balances IT topics.
- Consistent chat response format. Describe how answers become formatted SFT chat examples and why the consistent format matters.
- Full SFT configuration constraints. Explain why full SFT needs more explicit control of training settings than LoRA, and identify SFTConfig as the configuration class used here.
- Micro-batches and gradient accumulation. Explain how a per-device batch size of 1 and 16 gradient-accumulation steps provide an effective batch size of 16 without holding 16 examples’ activations at once.
- Conservative full-SFT learning rate. Justify a lower learning rate for full SFT than for LoRA using the section’s starting value and its explanation of forgetting.
- Full SFT Script Configuration. Explain how the script switches from LoRA to full SFT while retaining the same basic training workflow.
- Gradient Norm Warning Signs. Explain how to interpret the gradient norm pattern shown for this run and recognize signs of training instability.
- Initial response style check. Describe how to make an initial inference check after training: generate and read a few responses using the training chat template, looking for the intended step-by-step IT support style rather than generic chat.
- Validation loss and overfitting. Distinguish what falling, plateauing, and rising validation loss mean when training loss continues to fall, and describe when to stop training.
- Gradient norm stability signals. Explain how the trajectory of gradient norm indicates stable optimization, an excessively high learning rate, or diminishing benefit from further training.
- Healthy Training Metric Patterns. Describe the four metrics to watch during this SFT run and the pattern the section identifies as healthy for each.
- Loss Curve Interpretation. Explain what a loss curve plots and why comparing training loss with validation loss is more informative than reading either curve alone.
- Convergence and Overfitting Patterns. Distinguish healthy convergence from overfitting using the two loss curves, and state when to stop an overfitting run.
- Underfitting and Instability Patterns. Distinguish underfitting from instability by their loss patterns and describe the causes or response given for each.
- Validation loss as primary signal. Explain why validation loss, rather than training loss, should be the primary signal when monitoring training.
- Need for rigorous evaluation. Explain why fine-tuning alone does not establish that a model is better: evaluation is needed to tell whether it improved, degraded, or memorized the training data.
- Three evaluation purposes. Explain that evaluation checks whether fine-tuning taught the target behavior, preserved prior capabilities, and produced a model ready for production use.
- Complementary quantitative metrics. Explain why SFT evaluation should track both a task-specific correctness metric, such as accuracy or F1, and a generation-quality metric, such as perplexity or ROUGE-L.
- Base model baseline comparison. Explain why the base and fine-tuned models should be run on the same test set and compared side by side.
- Base and SFT comparison. Describe how the evaluation harness compares the base and full-SFT models on the same held-out questions, and explain what changes when it is used with a full checkpoint rather than a LoRA adapter.
- Small score changes and noise. Explain why the small overall Token-F1 lift in this IT-support run does not establish a substantial performance gain or, by itself, justify full SFT's added cost.
- Behavior beyond word overlap. Explain why Token-F1 can miss improvements in open-ended IT answers and describe the additional checks that revealed what SFT changed in this run.
- Safety and capability regression checks. Explain why benign-question refusal scores cannot establish safety retention, and describe how targeted behavioral tests check safety, knowledge, and formatting before deployment.
- Category Results and Next Steps. Describe how to respond when accuracy plateaus across categories versus when some categories improve and others regress. Explain that changes should be followed by re-evaluation rather than treating the first run as the final model.
- Baseline and Regression Checks. Explain how to judge a fine-tuned model against its base model using success thresholds set before training, and why behavioral tests are also needed.
- SFT Value Over LoRA. Explain which tasks are most likely to benefit from full-parameter SFT rather than LoRA, and why SFT’s greater capacity matters for each.
- SFT and LoRA performance gap. Explain that full SFT usually improves only modestly over a well-tuned LoRA on domain tasks, with a larger gap mainly for generation requiring novel output formats. Distinguish that comparison from claims of large gains over an untuned base model.
- Training data and SFT advantage. Explain how data quality and volume affect whether full SFT beats LoRA, including why small datasets can favor LoRA and when SFT’s advantage tends to emerge.
- Fine-tuning’s behavior boundary. Explain that LoRA and full SFT reshape how a model uses knowledge acquired during pre-training, such as following a preferred format or convention, but do not reliably give it facts absent from pre-training.
- Knowledge and behavior gap routing. Distinguish a missing-facts problem from an incorrect-or-inconsistent-presentation problem, and describe how RAG and fine-tuning address them separately in the legal contract review example.
- SFT and LoRA trade-off. Explain when the extra accuracy from full SFT might justify its additional compute, memory, and operational complexity compared with a well-tuned LoRA adapter. State why example figures should not be treated as universal.
- Training memory tradeoff. Explain why rank-16 LoRA needs less GPU memory than full SFT and what gradient checkpointing trades off.
- Checkpoint and deployment footprint. Describe how LoRA and full SFT differ in checkpoint size and in serving multiple variants, including the disk-space implication of an SFT run.
- Catastrophic Forgetting Risk. Explain how fine-tuning on a narrow dataset can erase previously learned capabilities and why full SFT poses more of this risk than LoRA.
- Forgetting Risk Defenses. Describe the measures this section recommends to reduce catastrophic forgetting during full SFT.
- SFT Data Quality and Volume. Explain why SFT data needs depend on task complexity and quality rather than a fixed example count, and why published volume ranges are starting points for experimentation.
- Enterprise Data Preparation. Describe how noisy enterprise records become chat-structured SFT examples and why curation precedes JSONL conversion.
- Assistant-Turn Training Loss. Explain which conversation tokens contribute to SFT loss and why assistant responses must consistently model the desired output.
- Production Ready Training Responses. Explain why SFT training responses should be correct, complete, and consistent in format and terminology.
- Representative Input Coverage. Describe how input diversity and a representative mix of difficulty help an SFT model handle its target domain.
- Curated Dataset Planning. Describe a practical size, format, and mix of sources for a moderate full-SFT task, and explain why curation matters more than source proportions or raw count.
- Full SFT benefits and risks. Explain why full SFT can learn a task more fully while also increasing the risk of forgetting, overfitting, or subtle failures.
- Preventive SFT training habits. Describe the routine checks and conservative defaults this section recommends to prevent common SFT failures.
- Catastrophic Forgetting Risks. Explain why catastrophic forgetting is a particular concern with full SFT and describe its two forms: loss of capability outside the training domain and regression in rejecting harmful requests.
- Forgetting Mitigation And Checks. Describe the section’s practical choices for reducing forgetting—gentle updates, limited epochs, and a mix of general-purpose examples—and explain how before-and-after testing checks for lost capabilities.
- Overfitting and loss gap. Explain how overfitting makes a fine-tuned model perform well on training examples but poorly on new inputs, and describe the training-versus-validation loss pattern that signals it.
- Early stopping at best checkpoint. Explain why training should stop when validation loss no longer improves and why the best checkpoint is preferable to the final one.
- Data diversity and model capacity. Explain why more diverse examples can reduce overfitting and when switching from full SFT to LoRA may help.
- Production distribution mismatch. Explain how an SFT model can perform well on training and validation data yet underperform after deployment when production inputs differ.
- Production aligned training and testing. Describe how to reduce distribution mismatch and check performance before full deployment using production examples, edge cases, held-out production data, and a live-traffic comparison.
- Silent max_length truncation. Explain how a too-small max_length clips training conversations, why loss and validation metrics may still look fine, and how the problem appears in generated answers.
- Pretraining configuration checks. Describe the three habits for catching configuration mistakes: measure templated example lengths and set max_length at least to the 95th percentile, log settings at job start, and run a short smoke test while inspecting sample generations.
- Ordered hyperparameter tuning sequence. Describe the tuning order after a conservative starting configuration fails to meet the target: learning rate, epochs, effective batch size, then warmup and weight decay if needed. Explain how validation loss and run logs guide comparisons.
- Tuning stop and data pivot. State when to stop hyperparameter tuning and explain why a plateau or exhausted sweep points toward improving the training data rather than continuing configuration changes.
- SFT single-target limitation. Explain why SFT alone may leave a model unable to choose the better of several acceptable responses to the same prompt.
- Preference optimization transition. Describe when preference optimization is the next step after SFT and how its training signal differs from SFT's.
- SFT and preference optimization stages. Explain what SFT and preference optimization each contribute, and why SFT comes first. Describe how the DeepSeek-R1 example illustrates this order.
- Reproducible SFT artifacts. State which four items to save to reproduce the SFT stage: the full model checkpoint, tokenizer files, SFTConfig, and evaluation report JSON.
- DPO evaluation baseline. Describe which SFT evaluation results to record as the baseline for measuring DPO improvements and where to keep them.
- SFT platform choice trade-off. Explain that enterprise teams may evaluate managed platforms before owning infrastructure, and that platform choices differ in operational simplicity and fine-grained control. Identify the three representative starting points: a managed API (OpenAI/Azure), a cloud ML platform (Vertex AI), and a self-hosted open-source approach (Hugging Face).
- Managed SFT workflow. Describe how to start a managed SFT job by uploading chat-completion JSONL data and making an API call, while the platform handles hyperparameter selection, checkpointing, and evaluation.
- Azure route and model availability. Explain why the section recommends Azure OpenAI for new projects and why its current SFT model list should be checked before training.
- Vertex AI tuning pathways. Distinguish managed supervised fine-tuning for supported Gemini models from custom training jobs for open-weight models, where you provide the script and Vertex provisions GPU infrastructure.
- Self-hosting control and responsibility. Explain that self-hosting offers maximum control but makes the team responsible for infrastructure management. Identify the chapter’s SFTTrainer setup as the API pattern referenced for this pathway.
- Training Data Privacy Guarantees. Distinguish where training data resides and what guarantees the section attributes to self-hosting, Azure and GCP, and OpenAI’s direct API.
- Privacy First Platform Selection. Apply the section's platform-selection rule, including when greater control justifies self-hosting and what maintenance it entails.
- Clinical case study framing. Describe the case study’s intended use and explain why its figures should not be read as results from a single deployment.
- Clinical dataset coverage. Describe the source and breadth of the selected clinical examples: 5,200 examples from a medical reasoning collection, spanning general medical knowledge, clinical reasoning, drug interactions, and patient communication.
- Conversation formatting and quality filtering. Explain how each example was structured as a clinical-assistant conversation and how the team screened examples for quality.
- Validation and temporal test sets. Distinguish the held-out validation set from the separate test set and explain why the test examples came from a different time period.
- LoRA and SFT comparison. Explain why the team tried LoRA before comparing it with full SFT: to determine whether full SFT’s additional capacity was warranted.
- Full SFT training configuration. Describe the main settings of the full SFT run, including the Qwen3-4B-Instruct base model, 4,680 examples after filtering and holdout, three epochs, and an effective batch size of 32 from a batch size of 4 with gradient accumulation of 8.
- Uneven gains across categories. Explain why drug interactions and clinical reasoning gained more than general medical knowledge after SFT.
- Patient communication register. Explain how plain-language training examples changed the model’s patient-facing responses.
- Human versus compute costs. Explain how the case study separates compute spend from staff time and why data curation and evaluation dominate the total investment.
- Clinical lookup time savings. Trace how using the fine-tuned model as a first-pass tool reduced estimated lookup time and how that per-query saving scales across the organization.
- ROI assumptions and caveats. Explain the estimated financial payoff and why actual value depends on adoption and ongoing maintenance.
- Quality of reasoning examples. Explain why filtering training examples mattered in the healthcare case: the model imitated the reasoning it was shown, not just final answers, and unfiltered examples led to contradictory reasoning.
- Safety regression testing. Explain the safety regression found after fine-tuning and how the team responded to it.
- Full lifecycle effort and costs. Explain why the case study recommends planning for the full adaptation lifecycle rather than only the training run, including recurring costs.
- Staged adaptation decision. Explain why teams should begin with lower-cost adaptation methods and consider full SFT only after simpler approaches, including LoRA, fall measurably short.
- Full SFT readiness conditions. Describe the quality, data, and maintenance conditions for considering full SFT, and what to do when LoRA falls short but curated data is insufficient.
- Net value of SFT. Apply the section's net-value calculation by comparing SFT's quality improvement against the best non-SFT alternative, valuing that improvement for the use case, and accounting for training, maintenance, and opportunity costs.
- Ongoing model maintenance. Explain why a fine-tuned model needs ongoing attention and how data drift, base model updates, and a reusable evaluation suite shape maintenance decisions.
- Lifecycle cost in SFT decisions. Justify including ongoing maintenance in an SFT ROI calculation and choosing a simpler approach when it meets the quality bar.
- Unintended behavioral regression risk. Explain why a model that improves in one area still needs regression checks before deployment.
- Relative safety regression checks. Explain how the safety suite compares the base and fine-tuned models on the same prompts and flags a drop of more than 10 percentage points in any tested category.
- Helpfulness-induced refusal regression. Describe the harmful-request refusal regression in the example and explain how training on exclusively confident, helpful answers can cause it.
- Limits of average validation loss. Explain why monitoring validation loss during SFT is necessary but insufficient: it averages performance across a held-out set, while failures may be concentrated in particular kinds of inputs.
- Targeted validation sets and signals. Distinguish in-distribution, out-of-distribution, and baseline capability validation by the examples each uses and the failure its degradation signals.
- Development and Deployment Test Suites. Distinguish the quick behavioral spot-check used after each training iteration from the broader regression suite used before shipping. Describe what the pre-deployment suite compares and reports.
- Validation Loss Early Stopping. Explain that early stopping based on validation loss is a straightforward way to prevent overfitting during SFT.
- General Purpose Data Replay. Describe how mixing general-purpose instruction-following examples with domain data reduces behavioral regression while preserving specialization.
- Distinct Regression Deployment Checks. Distinguish the effects of overfitting and behavioral regression, and explain why target metrics alone cannot establish deployment readiness.
What it builds on
- LoRA core idea (chapter 5) · Introduces LoRA’s adapter-based approach.
- When LoRA hits its limits (chapter 5) · Discusses where LoRA reaches its limits.
- Hands-on: Your first LoRA fine-tuning (chapter 5) · The earlier LoRA fine-tune used an IT support assistant.
- Data acquisition strategies (chapter 3) · Ways to acquire task-relevant training data.
- Diversity across user types, conditions, and variations (chapter 3) · The importance of diversity in training examples.
- Golden test sets and held-out evaluation (chapter 3) · The purpose of held-out evaluation data.
- Step 2: Train LoRA adapter (chapter 5) · The earlier LoRA adapter training workflow.
- Step 5: Inference (chapter 5) · The LoRA base-plus-adapter inference pattern.
Section by section
Introduction
Introduces full supervised fine-tuning as a more expressive but costlier option when a LoRA adapter falls short of a measured quality target.
- Full SFT versus LoRA updates: Explain how full SFT differs from LoRA in which model parameters it trains and why that gives SFT more freedom to reshape behavior.
Why it matters: This difference explains why full SFT may change behavior that an adapter cannot change enough. - SFT escalation and costs: Justify when the section recommends trying full SFT instead of continuing with LoRA, and describe the associated training costs and risk.
Why it matters: The quality gain must justify the extra resources and training risk.
Key terms: supervised fine-tuning (SFT), LoRA, low-rank constraint, network parameter, quality target, LoRA rank, plateaued, GPU memory, overfitting
6.1 Hands-on: SFT training workflow
Introduces a full-model supervised fine-tuning workflow for an IT support assistant and contrasts it with the earlier LoRA workflow.
- Full Model SFT Updates: Explain how this SFT workflow differs from LoRA: it omits the LoRA configuration and updates the full model rather than training an adapter, aiming to make support behavior, step-by-step formatting, and domain vocabulary the model’s defaults.
Why it matters: This distinction explains what changes when moving from the chapter 5 adapter to full-model SFT. - Comparable IT Support Example: Describe why the IT support assistant provides a direct comparison with chapter 5’s LoRA adapter: it uses the same kind of task and maps onto the earlier IT support dataset.
Why it matters: A shared task and dataset make the two adaptation approaches easier to compare.
Key terms: supervised fine-tuning, LoRA configuration, SFTTrainer, low-rank adapters, IT technical support assistant, IT support dataset, LoRA adapter
6.1.1 Data preparation
This section prepares supervised fine-tuning data by sourcing real IT questions and answers, balancing them with general examples, and converting them into a consistent chat format.
- Real IT question sourcing: Describe how the pipeline obtains real IT question-and-answer pairs and selects a reference response.
Why it matters: Starting with real questions and vetted answers makes the training examples relevant to the support task. - Balanced training example mix: Explain why the pipeline mixes general-purpose examples into the IT data and balances IT topics.
Why it matters: The mix and topic balance shape what the model practices without letting one style or topic take over. - IT focused evaluation separation: Explain how the validation and test data are kept focused on the intended IT-support behavior.
Why it matters: A separate IT test set gives a more meaningful check of the support behavior the model is meant to deliver. - Consistent chat response format: Describe how answers become formatted SFT chat examples and why the consistent format matters.
Why it matters: Consistent examples teach the model the response structure and persona expected in the target use case.
Key terms: Stack Exchange, reference response, preference pairs, Dolly, 80/20 mix, stratify, fixed seed, validation set, held-out test set, model-selection
6.1.2 Model loading and training configuration
This section explains how to configure full supervised fine-tuning conservatively to manage memory use, protect pretrained behavior, and catch problems before a full run.
- Full SFT configuration constraints: Explain why full SFT needs more explicit control of training settings than LoRA, and identify SFTConfig as the configuration class used here.
Why it matters: The reason for these settings helps you choose a safe starting configuration rather than copying numbers blindly. - Micro-batches and gradient accumulation: Explain how a per-device batch size of 1 and 16 gradient-accumulation steps provide an effective batch size of 16 without holding 16 examples’ activations at once.
Why it matters: This lets a full-SFT run use a useful effective batch size when model-wide optimizer states strain memory. - Gradient checkpointing trade-off: Describe how gradient checkpointing reduces activation memory during full SFT and what it costs.
Why it matters: The memory-for-compute trade-off can make full SFT fit on a consumer GPU. - Conservative full-SFT learning rate: Justify a lower learning rate for full SFT than for LoRA using the section’s starting value and its explanation of forgetting.
Why it matters: A learning rate suitable for an adapter may be too aggressive when every model parameter changes. - Warmup and weight decay: Explain the separate stability and overfitting purposes of the section’s warmup and weight-decay settings.
Why it matters: These settings address different risks: an unstable start and excessive fitting to the training examples. - Conservative rollout and smoke test: Describe the recommended way to begin and check a full-SFT run before committing to longer training.
Why it matters: A quick check can expose configuration or data problems before they waste a longer training run.
Key terms: full SFT, LoRA, SFTConfig, optimizer states, activations, catastrophic forgetting, per_device_train_batch_size, gradient_accumulation_steps, micro-batch, effective batch size
6.1.3 Training script
This section explains how the SFT training script differs from LoRA, what full-model training costs, and how to interpret the run’s initial training signals.
- Full SFT Script Configuration: Explain how the script switches from LoRA to full SFT while retaining the same basic training workflow.
Why it matters: A small configuration change makes the trainer update the entire model rather than a LoRA adapter. - Full SFT Resource Costs: Describe the practical resource consequences of training all parameters rather than a LoRA adapter.
Why it matters: Similar-looking scripts can have very different hardware and storage requirements. - Gradient Norm Warning Signs: Explain how to interpret the gradient norm pattern shown for this run and recognize signs of training instability.
Why it matters: Gradient norm offers a quick check that training is progressing rather than diverging or stalling. - Validation Loss Checkpoint Selection: Explain what the reported validation losses imply and which checkpoint the script selects.
Why it matters: The final checkpoint is chosen by validation performance, not simply by which epoch ran last.
Key terms: SFTTrainer, peft_config, full SFT, learning rate, full-model checkpoints, GPU memory usage, GPU time, LoRA, gradient norm, NaN
6.1.4 Inference
This section shows how to load a fully fine-tuned model and make an initial qualitative check of its responses.
- Initial response style check: Describe how to make an initial inference check after training: generate and read a few responses using the training chat template, looking for the intended step-by-step IT support style rather than generic chat.
Why it matters: Reading a few outputs helps you spot whether the model is behaving in the style you trained it to use. - Full SFT model loading: Explain why inference loading is simpler for full SFT than for LoRA: the output directory contains the complete model, so a single from_pretrained call loads it without a separate base model and adapter.
Why it matters: Knowing what the training output contains tells you what to load for inference.
Key terms: chat template, step-by-step IT support style, generic chat, full SFT, from_pretrained, output directory, adapter deltas
6.2 Monitoring training
This section explains how training loss, validation loss, and gradient norm reveal whether an SFT run is learning, overfitting, or becoming unstable.
- Training loss trajectory: Describe what training loss measures and how it typically changes during a healthy SFT run.
Why it matters: Training loss gives you an early signal of progress, but it cannot by itself show whether the model will generalize. - Validation loss and overfitting: Distinguish what falling, plateauing, and rising validation loss mean when training loss continues to fall, and describe when to stop training.
Why it matters: Validation loss helps you avoid continuing a run that improves on training examples while getting worse on new inputs. - Gradient norm stability signals: Explain how the trajectory of gradient norm indicates stable optimization, an excessively high learning rate, or diminishing benefit from further training.
Why it matters: Watching the trend helps you distinguish harmless startup spikes from persistent optimization problems.
Key terms: training loss, epoch, validation loss, generalization, overfitting, checkpoint, gradient norm, grad_norm, learning rate
6.2.1 Tools for monitoring training
This section shows how to view SFT training metrics with TensorBoard or W&B and what a healthy run looks like across four metrics.
- Healthy Training Metric Patterns: Describe the four metrics to watch during this SFT run and the pattern the section identifies as healthy for each.
Why it matters: These patterns give you a quick way to judge whether the monitored run looks healthy. - Monitoring Tool Tradeoffs: Distinguish TensorBoard from W&B as ways to visualize Hugging Face Trainer metrics.
Why it matters: You can choose local monitoring without an account or use cloud features when comparing and sharing runs. - Monitoring Setup Options: Describe the configuration given for viewing TensorBoard metrics and the options given for using or disabling W&B.
Why it matters: The configuration lets you view metrics during training without making W&B a dependency.
Key terms: training loss, validation loss, gradient norm, learning rate schedule, TensorBoard, Weights & Biases, W&B, Hugging Face Trainer, report_to, logging_dir
6.2.2 Reading the loss curves
This section shows how training and validation loss curves, together with gradient-norm warnings, reveal whether an SFT run is learning, overfitting, underfitting, or unstable.
- Loss Curve Interpretation: Explain what a loss curve plots and why comparing training loss with validation loss is more informative than reading either curve alone.
Why it matters: Comparing the curves gives you a quick way to judge a training run’s health. - Convergence and Overfitting Patterns: Distinguish healthy convergence from overfitting using the two loss curves, and state when to stop an overfitting run.
Why it matters: A falling training loss is not a reason to keep training once validation performance deteriorates. - Underfitting and Instability Patterns: Distinguish underfitting from instability by their loss patterns and describe the causes or response given for each.
Why it matters: These patterns call for different investigations rather than simply letting the run continue. - Training Log Health Checks: Describe how the training-log summary uses validation-loss trends and gradient norms, and how to respond to a gradient-norm warning in context.
Why it matters: Reading warnings alongside the loss trend helps you act on genuine instability without overreacting to a transient alert. - Worked Run Loss Gap: Explain what the example run’s falling training loss, flat validation loss, and brief gradient-norm warning imply.
Why it matters: The worked run shows why improving training loss should not be mistaken for an improvement on held-out examples.
Key terms: loss curve, training loss, validation loss, training steps, epochs, healthy convergence, overfitting, epoch, underfitting, instability
6.2.3 Red flags and concrete thresholds
This section gives a primary signal for monitoring fine-tuning and cautions that its referenced thresholds are starting points, not fixed rules.
- Validation loss as primary signal: Explain why validation loss, rather than training loss, should be the primary signal when monitoring training.
Why it matters: A falling training loss alone does not show that training is helping. - Thresholds as contextual starting points: Describe how to treat the referenced training thresholds and the model and dataset sizes for which they are offered.
Why it matters: The stated context helps you avoid treating a suggested threshold as a universal cutoff.
Key terms: validation loss, training loss, actionable thresholds, 4B to 7B models, 400 to 1,000 examples
6.3 Evaluating the fine-tuned model
This section explains why fine-tuning requires evaluation rather than an assumption that the model improved.
- Need for rigorous evaluation: Explain why fine-tuning alone does not establish that a model is better: evaluation is needed to tell whether it improved, degraded, or memorized the training data.
Why it matters: Evaluating the result keeps a fine-tuned model from being mistaken for an improved one.
Key terms: fine-tuning, rigorous evaluation, memorize the training data
6.3.1 Evaluation strategy
This section explains how to evaluate an SFT model using complementary metrics, human review, and a comparison with the base model.
- Three evaluation purposes: Explain that evaluation checks whether fine-tuning taught the target behavior, preserved prior capabilities, and produced a model ready for production use.
Why it matters: A gain on the target task alone does not establish that the model is ready to use. - Complementary quantitative metrics: Explain why SFT evaluation should track both a task-specific correctness metric, such as accuracy or F1, and a generation-quality metric, such as perplexity or ROUGE-L.
Why it matters: A fluent-sounding answer can still be wrong, while a correct one can be awkward or repetitive. - Qualitative response review: Describe how to complement metrics by reviewing responses to 20 to 30 hand-picked prompts covering important behaviors after every training run.
Why it matters: Reading representative responses can reveal problems that aggregate scores miss. - Base model baseline comparison: Explain why the base and fine-tuned models should be run on the same test set and compared side by side.
Why it matters: Without a baseline, a strong result may not represent an improvement—and a lost capability may go unnoticed.
Key terms: target behavior, prior capabilities, production use, quantitative metrics, accuracy, F1, perplexity, ROUGE-L, qualitative review, hand-picked prompts
6.3.2 Hands-on evaluation code
This section evaluates full SFT against a base model using held-out scores and behavioral tests, then shows why small Token-F1 changes can conceal meaningful changes in output format and safety.
- Base and SFT comparison: Describe how the evaluation harness compares the base and full-SFT models on the same held-out questions, and explain what changes when it is used with a full checkpoint rather than a LoRA adapter.
Why it matters: A consistent harness helps you attribute score differences to the models rather than to different tests or evaluation setups. - Small score changes and noise: Explain why the small overall Token-F1 lift in this IT-support run does not establish a substantial performance gain or, by itself, justify full SFT's added cost.
Why it matters: A small gain on a small test set is a poor basis for paying substantially more to train and run a model. - Category score swings: Distinguish a category score swing caused by test composition and varied answer phrasing from a regression that may call for targeted training data.
Why it matters: Diagnosing a category swing before adding data avoids treating noisy word overlap as a training-data shortage. - Behavior beyond word overlap: Explain why Token-F1 can miss improvements in open-ended IT answers and describe the additional checks that revealed what SFT changed in this run.
Why it matters: Measuring the behavior you trained for prevents a word-overlap metric from hiding a useful improvement. - Safety and capability regression checks: Explain why benign-question refusal scores cannot establish safety retention, and describe how targeted behavioral tests check safety, knowledge, and formatting before deployment.
Why it matters: A model can score acceptably on ordinary task questions yet lose protections that must be checked before release.
Key terms: held-out test set, Token-F1, exact match, safety refusal rate, full checkpoint, greedy decoding, held-out test split, evaluation noise, GPU memory, category-level breakdown
6.3.3 Making decisions from evaluation results
This section shows how to turn fine-tuning evaluation results into training changes and decide whether a model is ready to ship.
- Category Results and Next Steps: Describe how to respond when accuracy plateaus across categories versus when some categories improve and others regress. Explain that changes should be followed by re-evaluation rather than treating the first run as the final model.
Why it matters: Different patterns in the results point to different training changes, and those changes need to be checked again. - Baseline and Regression Checks: Explain how to judge a fine-tuned model against its base model using success thresholds set before training, and why behavioral tests are also needed.
Why it matters: A metric alone may not show whether fine-tuning improved the base model or introduced a behavioral regression.
Key terms: accuracy plateaus, underperforming category, upsampling, re-evaluating, fine-tuned model, base model, success thresholds, behavioral tests, regressions
6.4 When SFT delivers value
This section identifies the tasks where full-parameter SFT can deliver more value than LoRA and explains why.
- SFT Value Over LoRA: Explain which tasks are most likely to benefit from full-parameter SFT rather than LoRA, and why SFT’s greater capacity matters for each.
Why it matters: These distinctions help you decide when the added scope of full-parameter tuning is justified instead of using LoRA.
Key terms: SFT, LoRA, low-rank constraint, rigid schema, domain-specific reasoning chains, output consistency
6.4.1 Realistic expectations
This section sets realistic expectations for how much full SFT improves on a well-tuned LoRA adapter and when training data makes that improvement likely.
- SFT and LoRA performance gap: Explain that full SFT usually improves only modestly over a well-tuned LoRA on domain tasks, with a larger gap mainly for generation requiring novel output formats. Distinguish that comparison from claims of large gains over an untuned base model.
Why it matters: Comparing SFT with a strong LoRA baseline prevents exaggerated expectations of its gains. - Training data and SFT advantage: Explain how data quality and volume affect whether full SFT beats LoRA, including why small datasets can favor LoRA and when SFT’s advantage tends to emerge.
Why it matters: Dataset size and quality help determine whether full SFT is likely to repay its added complexity.
Key terms: full SFT, well-tuned LoRA, vanilla base model, novel output formats, data quality, data volume, overfitting, SFT, LoRA
6.5 Fine-tuning refines, doesn’t inject knowledge
This section explains why standard fine-tuning is best used to change model behavior rather than supply missing facts, and how to pair it with RAG when a task needs both.
- Fine-tuning’s behavior boundary: Explain that LoRA and full SFT reshape how a model uses knowledge acquired during pre-training, such as following a preferred format or convention, but do not reliably give it facts absent from pre-training.
Why it matters: It prevents you from treating a training run as a reliable way to add missing facts. - Knowledge and behavior gap routing: Distinguish a missing-facts problem from an incorrect-or-inconsistent-presentation problem, and describe how RAG and fine-tuning address them separately in the legal contract review example.
Why it matters: Separating these gaps helps keep changing information current without sacrificing consistent outputs. - Self-distillation research caveat: Describe the reported SDFT result as a possible exception to the usual behavior-versus-knowledge boundary, while stating why the section still recommends that boundary as the default for enterprise SFT.
Why it matters: It keeps the practical rule accurate without treating emerging research as routine enterprise practice.
Key terms: pre-training, fine-tuning, LoRA, full SFT, knowledge gap, behavior gap, retrieval-augmented generation (RAG), risk-rating taxonomy, Self-Distillation Fine-Tuning (SDFT), standard SFT
6.6 SFT vs. LoRA: Realistic performance expectations
This section frames the decision between full SFT and LoRA as a trade-off between accuracy gains and added resource and operational costs.
- SFT and LoRA trade-off: Explain when the extra accuracy from full SFT might justify its additional compute, memory, and operational complexity compared with a well-tuned LoRA adapter. State why example figures should not be treated as universal.
Why it matters: It keeps the comparison focused on whether an accuracy gain is worth the practical cost, rather than on accuracy alone.
Key terms: full SFT, LoRA adapter, compute, memory, operational complexity
6.6.1 The cost-performance tradeoff
This section compares the memory and deployment costs of rank-16 LoRA with full-parameter SFT and recommends trying a higher LoRA rank before switching when performance is close to the target.
- Training memory tradeoff: Explain why rank-16 LoRA needs less GPU memory than full SFT and what gradient checkpointing trades off.
Why it matters: Memory needs can determine whether full SFT is practical on your available GPU. - Checkpoint and deployment footprint: Describe how LoRA and full SFT differ in checkpoint size and in serving multiple variants, including the disk-space implication of an SFT run.
Why it matters: The size difference affects both storage planning and the cost of serving several model variants. - Higher rank before full SFT: Apply the section's recommendation when a LoRA adapter comes close to the target metric.
Why it matters: A near-target adapter may be improved without immediately taking on full SFT's costs.
Key terms: LoRA at rank 16, full SFT, optimizer states, gradient checkpointing, LoRA adapter, full SFT checkpoint, shared base model, save_total_limit, target metric, LoRA rank
6.6.2 The forgetting risk (catastrophic forgetting)
This section explains why full SFT is more vulnerable to catastrophic forgetting than LoRA, how to reduce that risk, and why the risk matters when weighing SFT’s modest accuracy advantage against its costs.
- Catastrophic Forgetting Risk: Explain how fine-tuning on a narrow dataset can erase previously learned capabilities and why full SFT poses more of this risk than LoRA.
Why it matters: Improving performance on a narrow task can come at the expense of capabilities the base model already had. - Forgetting Risk Defenses: Describe the measures this section recommends to reduce catastrophic forgetting during full SFT.
Why it matters: These choices help preserve broader capabilities while adapting the model to a narrower task. - SFT Lifecycle Tradeoff: State how the section weighs SFT’s accuracy advantage against its operational costs when deciding whether to use it.
Why it matters: A modest accuracy gain may not justify the costs of operating a fully fine-tuned model.
Key terms: catastrophic forgetting, full SFT, LoRA, general-purpose skills, safety behaviors, learning rate, early stopping, validation loss, general-purpose examples, rehearsal
6.7 Data requirements
This section explains how data quality, task complexity, preparation, and assistant-turn supervision shape SFT data requirements.
- SFT Data Quality and Volume: Explain why SFT data needs depend on task complexity and quality rather than a fixed example count, and why published volume ranges are starting points for experimentation.
Why it matters: It helps you plan a dataset without mistaking an example-count target for a quality guarantee. - Low-Data LoRA Choice: Explain why LoRA is often preferable below the minimum viable SFT data threshold and what examples are needed to shift model behavior.
Why it matters: It gives you a reason to reconsider full SFT when representative training data is scarce. - Enterprise Data Preparation: Describe how noisy enterprise records become chat-structured SFT examples and why curation precedes JSONL conversion.
Why it matters: It keeps formatting a training file from being mistaken for preparing training-ready data. - Assistant-Turn Training Loss: Explain which conversation tokens contribute to SFT loss and why assistant responses must consistently model the desired output.
Why it matters: It explains why carefully written assistant answers matter even when the surrounding prompts are well prepared.
Key terms: SFT, task complexity, data quality, minimum viable, LoRA, natural regularization, representative, diverse examples, chat-structured JSONL, personally identifiable information, curation
6.7.1 Quality over quantity
This section explains why full SFT benefits more from carefully curated, representative examples than from simply collecting more data, and gives practical guidance for assembling and splitting a dataset.
- Production Ready Training Responses: Explain why SFT training responses should be correct, complete, and consistent in format and terminology.
Why it matters: A fine-tune can reproduce flaws in the answers it was trained to imitate. - Representative Input Coverage: Describe how input diversity and a representative mix of difficulty help an SFT model handle its target domain.
Why it matters: Varied, realistic examples help the model respond to more than the narrow patterns it has seen repeatedly. - Held Out Validation Split: State how the section says to divide examples between training and validation and protect the validation set.
Why it matters: A genuinely held-out, task-balanced set provides a separate check on the fine-tuned model. - Curated Dataset Planning: Describe a practical size, format, and mix of sources for a moderate full-SFT task, and explain why curation matters more than source proportions or raw count.
Why it matters: This gives a workable dataset plan without treating example count as a substitute for review. - Model Size Data Scaling: Explain the section's approximate relationship between model size and the number of SFT examples needed for a comparable relative improvement.
Why it matters: Dataset-size estimates should account for the starting model rather than be treated as universal rules.
Key terms: correct and complete assistant responses, consistent formatting and conventions, contradictory training signal, diverse inputs within the target domain, representative difficulty distribution, edge cases, training/validation set, stratified by task type, held out, full SFT
6.8 Common pitfalls and how to avoid them
This section explains why full SFT increases both learning capacity and failure risk, then gives preventive habits for training runs.
- Full SFT benefits and risks: Explain why full SFT can learn a task more fully while also increasing the risk of forgetting, overfitting, or subtle failures.
Why it matters: The same flexibility that makes full SFT useful makes its failures important to watch for. - Preventive SFT training habits: Describe the routine checks and conservative defaults this section recommends to prevent common SFT failures.
Why it matters: These checks and defaults provide a practical starting point for reducing preventable training failures.
Key terms: Full SFT, forget, overfit, smoke test, safety regression check, early stopping, low learning rate, few epochs, weight decay
6.8.1 Avoiding catastrophic forgetting
This section explains how full SFT can cause catastrophic forgetting and presents practical ways to reduce and check that risk.
- Catastrophic Forgetting Risks: Explain why catastrophic forgetting is a particular concern with full SFT and describe its two forms: loss of capability outside the training domain and regression in rejecting harmful requests.
Why it matters: A domain improvement is not enough if the model loses useful abilities or stops refusing harmful requests. - Forgetting Mitigation And Checks: Describe the section’s practical choices for reducing forgetting—gentle updates, limited epochs, and a mix of general-purpose examples—and explain how before-and-after testing checks for lost capabilities.
Why it matters: These choices help preserve the base model’s broader usefulness and make losses visible rather than assuming training was harmless. - On Policy Alternative Tradeoff: Describe the reported comparison between standard SFT and Self-Distillation Fine-Tuning as evidence that forgetting may be reduced, while noting the alternative’s compute and model-capability requirements.
Why it matters: The comparison shows that standard SFT’s forgetting is not necessarily unavoidable, but reducing it can demand more resources.
Key terms: catastrophic forgetting, full SFT, LoRA, capability loss, safety regression, learning rate, epochs, general-purpose examples, rehearsal mechanism, golden set
6.8.2 Overfitting
This section explains how to recognize overfitting in supervised fine-tuning and reduce it through early stopping, regularization, more diverse data, or lower training capacity.
- Overfitting and loss gap: Explain how overfitting makes a fine-tuned model perform well on training examples but poorly on new inputs, and describe the training-versus-validation loss pattern that signals it.
Why it matters: Training loss alone can look encouraging even when the model is getting worse at handling new requests. - Early stopping at best checkpoint: Explain why training should stop when validation loss no longer improves and why the best checkpoint is preferable to the final one.
Why it matters: The last training epoch is not necessarily the one that performs best on unseen inputs. - Regularization against memorization: Describe how weight decay or dropout can make it harder for an SFT model to memorize individual examples.
Why it matters: Regularization offers a way to curb memorization without relying solely on when training stops. - Data diversity and model capacity: Explain why more diverse examples can reduce overfitting and when switching from full SFT to LoRA may help.
Why it matters: A persistent loss gap may call for changing the data or training approach, not just training the same model longer.
Key terms: overfitting, training loss, validation loss, transferable patterns, early stopping, best checkpoint, regularization, weight decay, dropout, attention layers
6.8.3 Distribution mismatch
This section explains why supervised fine-tuning can fail when curated training examples differ from production inputs, and how to detect and reduce that mismatch.
- Production distribution mismatch: Explain how an SFT model can perform well on training and validation data yet underperform after deployment when production inputs differ.
Why it matters: Healthy development metrics do not guarantee that the fine-tuned model will handle real user inputs. - Production aligned training and testing: Describe how to reduce distribution mismatch and check performance before full deployment using production examples, edge cases, held-out production data, and a live-traffic comparison.
Why it matters: These checks make it more likely that improvements measured during development will hold up in production.
Key terms: distribution mismatch, training distribution, production distribution, validation-set metrics, production traffic, adversarial and edge-case examples, held-out set, A/B tests, task-specific metrics
6.8.4 Hyperparameter misconfiguration
This section explains how a silently misconfigured max_length can damage SFT outputs and gives three checks to catch configuration problems before a full training run.
- Silent max_length truncation: Explain how a too-small max_length clips training conversations, why loss and validation metrics may still look fine, and how the problem appears in generated answers.
Why it matters: A healthy-looking loss curve does not prove the model saw complete answers during training. - Pretraining configuration checks: Describe the three habits for catching configuration mistakes: measure templated example lengths and set max_length at least to the 95th percentile, log settings at job start, and run a short smoke test while inspecting sample generations.
Why it matters: These quick checks can prevent a long training run with settings or data that produce unusable answers.
Key terms: max_length, truncation, loss, validation metrics, chat template, 95th percentile, hyperparameters, smoke test, sample generations
6.9 Hyperparameter iteration strategy
This section gives a diagnosis-first, one-variable-at-a-time strategy for tuning SFT hyperparameters and deciding when to stop tuning and improve the data instead.
- Diagnosis before hyperparameter changes: Explain why loss-curve patterns should guide tuning before any settings change, and why runs should change only one variable at a time.
Why it matters: A run is more useful when you can tell what problem it addressed and what change affected the result. - Ordered hyperparameter tuning sequence: Describe the tuning order after a conservative starting configuration fails to meet the target: learning rate, epochs, effective batch size, then warmup and weight decay if needed. Explain how validation loss and run logs guide comparisons.
Why it matters: An ordered sweep focuses training runs on the settings most likely to help while preserving a clear comparison between runs. - Tuning stop and data pivot: State when to stop hyperparameter tuning and explain why a plateau or exhausted sweep points toward improving the training data rather than continuing configuration changes.
Why it matters: A stopping rule prevents spending more training runs on settings when the remaining improvement is more likely to come from data.
Key terms: underfitting, overfitting, loss-curve patterns, learning rate, epochs, effective batch size, gradient_accumulation_steps, warmup, weight decay, diminishing returns
6.10 Transitioning from SFT to preference optimization (DPO and RLHF)
This section explains when to move beyond SFT and how ranked responses provide the quality signal that SFT lacks.
- SFT single-target limitation: Explain why SFT alone may leave a model unable to choose the better of several acceptable responses to the same prompt.
Why it matters: This explains why more SFT is not necessarily the answer when responses are acceptable but still miss the desired quality bar. - Preference optimization transition: Describe when preference optimization is the next step after SFT and how its training signal differs from SFT's.
Why it matters: It gives you a way to decide whether an acceptable SFT model needs further work and what kind of feedback would address the gap.
Key terms: SFT, single target, quality distinctions, quality bar, preference optimization, DPO, GRPO, RLHF, pairs
6.10.1 The two-stage pattern
Explains why supervised fine-tuning (SFT) establishes a baseline before preference optimization refines response quality.
- SFT and preference optimization stages: Explain what SFT and preference optimization each contribute, and why SFT comes first. Describe how the DeepSeek-R1 example illustrates this order.
Why it matters: The order clarifies what each training stage is meant to accomplish and why preference tuning needs a capable starting model.
Key terms: SFT, cold start, preference optimization, preference signal, GRPO
6.10.2 Preparing your SFT checkpoint for preference optimization
The section explains what to preserve from SFT, how to record its evaluation baseline, and when to deploy the checkpoint or use it for preference optimization.
- Reproducible SFT artifacts: State which four items to save to reproduce the SFT stage: the full model checkpoint, tokenizer files, SFTConfig, and evaluation report JSON.
Why it matters: A complete set of saved artifacts makes the SFT stage available for later work. - DPO evaluation baseline: Describe which SFT evaluation results to record as the baseline for measuring DPO improvements and where to keep them.
Why it matters: Recording the baseline lets you compare later preference-optimized results against SFT. - SFT checkpoint next steps: Explain when to deploy the SFT checkpoint and when to use it as the starting point for preference optimization.
Why it matters: The SFT result may already be sufficient, so further optimization depends on the quality you need.
Key terms: full model checkpoint, tokenizer files, SFTConfig, evaluation report JSON, DPO baseline, overall Token-F1, per-category scores, safety refusal rate, eval report JSON, SFT checkpoint
6.11 Platform-specific implementation
This section frames enterprise SFT platform selection as a trade-off between operational simplicity and fine-grained control, using three representative starting points.
- SFT platform choice trade-off: Explain that enterprise teams may evaluate managed platforms before owning infrastructure, and that platform choices differ in operational simplicity and fine-grained control. Identify the three representative starting points: a managed API (OpenAI/Azure), a cloud ML platform (Vertex AI), and a self-hosted open-source approach (Hugging Face).
Why it matters: This framing helps you compare platform options without treating one implementation approach as the default.
Key terms: managed API, cloud ML platform, self-hosted open-source approach, operational simplicity, fine-grained control
6.11.1 OpenAI and Azure OpenAI
This section explains the managed SFT workflow on OpenAI and Azure OpenAI, and the platform and model-availability constraints that affect planning.
- Managed SFT workflow: Describe how to start a managed SFT job by uploading chat-completion JSONL data and making an API call, while the platform handles hyperparameter selection, checkpointing, and evaluation.
Why it matters: This clarifies what the managed service does for you and what you still need to provide. - Azure route and model availability: Explain why the section recommends Azure OpenAI for new projects and why its current SFT model list should be checked before training.
Why it matters: Platform access and model retirement dates can change whether a planned fine-tune is viable. - Managed platform controls and tradeoffs: Distinguish the training settings users can control from those the platform fixes, and describe the Azure features and discounted tiers mentioned in the section.
Why it matters: These limits and options help you decide whether managed SFT fits your control, infrastructure, and residency needs.
Key terms: SFT, JSONL, chat-completion format, checkpointing, Azure OpenAI, GPT-4.1 family, supervised fine-tuning, model list, epoch count, batch size
6.11.2 Google Vertex AI
This subsection distinguishes Vertex AI’s managed Gemini tuning from its custom training jobs and notes that Vertex handles LoRA configuration internally.
- Vertex AI tuning pathways: Distinguish managed supervised fine-tuning for supported Gemini models from custom training jobs for open-weight models, where you provide the script and Vertex provisions GPU infrastructure.
Why it matters: The model you want to tune determines which Vertex AI pathway you use and what you must supply. - Internal LoRA configuration: Explain that Vertex AI uses LoRA internally for efficient tuning, so users do not have to configure adapter parameters themselves.
Why it matters: It clarifies one tuning configuration task that Vertex AI takes off your hands.
Key terms: managed Gemini tuning pathway, custom training job pathway, open-weight models, LoRA, adapter parameters
6.11.3 Hugging Face self-hosted
This subsection outlines the control-versus-management tradeoff of self-hosted SFT and the infrastructure, distributed-training, and tracking choices it requires.
- Self-hosting control and responsibility: Explain that self-hosting offers maximum control but makes the team responsible for infrastructure management. Identify the chapter’s SFTTrainer setup as the API pattern referenced for this pathway.
Why it matters: The control you gain from self-hosting comes with operational responsibility. - Self-hosted deployment decisions: Describe the three decisions named for self-hosted deployment: where to obtain GPU infrastructure, whether to use distributed training for models above 7B parameters, and how to track experiments. State that DeepSpeed or FSDP can handle distributed training through SFTTrainer’s TrainingArguments integration, while the Trainer API supports the named tracking tools.
Why it matters: These choices turn self-hosting from a general preference into a deployment plan.
Key terms: self-hosted pathway, infrastructure management, SFTTrainer, Cloud VMs with GPUs, on-premises GPU servers, managed platforms, DeepSpeed, FSDP, TrainingArguments
6.11.4 Cost and privacy considerations
This section compares SFT platform costs and privacy guarantees, then gives a rule for choosing between managed and self-hosted training.
- SFT Platform Cost Tradeoffs: Explain how managed API and self-hosted SFT costs differ, and why training charges alone do not capture total cost.
Why it matters: The cheapest-looking training rate may not be the cheapest platform to operate. - Training Data Privacy Guarantees: Distinguish where training data resides and what guarantees the section attributes to self-hosting, Azure and GCP, and OpenAI’s direct API.
Why it matters: Privacy requirements can rule out a platform regardless of its training price. - Privacy First Platform Selection: Apply the section's platform-selection rule, including when greater control justifies self-hosting and what maintenance it entails.
Why it matters: This rule helps avoid taking on infrastructure work unless requirements warrant it.
Key terms: managed APIs, per training token, self-hosted infrastructure, per GPU-hour, TCO, self-hosted training, tenant-level isolation, enterprise agreement, HIPAA BAA, SOC 2
6.12 Real case study: Healthcare clinical Q&A
Introduces an illustrative clinical Q&A fine-tuning case study and clarifies that its figures represent published research outcomes, not one deployment.
- Clinical case study framing: Describe the case study’s intended use and explain why its figures should not be read as results from a single deployment.
Why it matters: This framing helps you interpret the later case-study figures without mistaking them for measurements from one deployed system.
Key terms: illustrative composite, medical question-answer pairs, clinical decision support, lookup and summarize
6.12.1 Dataset and preparation
This section describes how the clinical Q&A case study selected, formatted, cleaned, and split its data for supervised fine-tuning and evaluation.
- Clinical dataset coverage: Describe the source and breadth of the selected clinical examples: 5,200 examples from a medical reasoning collection, spanning general medical knowledge, clinical reasoning, drug interactions, and patient communication.
Why it matters: The mix of categories defines the clinical tasks represented in the case study. - Conversation formatting and quality filtering: Explain how each example was structured as a clinical-assistant conversation and how the team screened examples for quality.
Why it matters: Formatting establishes the behavior shown to the model, while filtering keeps flawed examples out of training. - Validation and temporal test sets: Distinguish the held-out validation set from the separate test set and explain why the test examples came from a different time period.
Why it matters: The separate, time-shifted test set helps make the case study’s evaluation less vulnerable to temporal leakage.
Key terms: FreedomIntelligence/medical-o1-reasoning-SFT, clinical reasoning, drug interactions, patient communication, multi-turn conversation, structured reasoning, deduplication, length filtering, stratified sample, validation set
6.12.2 Training configuration
The healthcare case study compares LoRA with full SFT and specifies the configuration used for the full SFT run.
- LoRA and SFT comparison: Explain why the team tried LoRA before comparing it with full SFT: to determine whether full SFT’s additional capacity was warranted.
Why it matters: The comparison establishes what the team needed to justify before choosing the more expressive approach. - Full SFT training configuration: Describe the main settings of the full SFT run, including the Qwen3-4B-Instruct base model, 4,680 examples after filtering and holdout, three epochs, and an effective batch size of 32 from a batch size of 4 with gradient accumulation of 8.
Why it matters: These settings make the case study’s full SFT run concrete enough to interpret alongside its results.
Key terms: LoRA, full SFT, additional capacity, Qwen3-4B-Instruct, training examples, epochs, gradient accumulation, effective batch size
6.12.3 Results
This section explains how the clinical Q&A model was evaluated and why supervised fine-tuning improved some categories more than others.
- Clinical response scoring rubric: Describe how clinical Q&A responses were scored against expert-verified reference answers for factual correctness, reasoning quality, and completeness on a 0–100 scale.
Why it matters: The scoring method clarifies what the reported improvements measure. - Uneven gains across categories: Explain why drug interactions and clinical reasoning gained more than general medical knowledge after SFT.
Why it matters: Different starting weaknesses explain why fine-tuning does not improve every category equally. - Patient communication register: Explain how plain-language training examples changed the model’s patient-facing responses.
Why it matters: This result shows how examples can shape communication style for a clinical audience.
Key terms: rubric-based evaluation, expert-verified reference answers, drug interactions, clinical reasoning, general medical knowledge, SFT, patient communication, plain-language explanations, clinical jargon, appropriate register
6.12.4 Cost breakdown
The section explains why staff time, rather than GPU compute, dominates the cost of fine-tuning the case-study model.
- Human versus compute costs: Explain how the case study separates compute spend from staff time and why data curation and evaluation dominate the total investment.
Why it matters: Budgeting only for GPU time would miss the main costs of this fine-tuning effort.
Key terms: compute spend, staff time, data curation, evaluation
6.12.5 ROI calculation
This section estimates the clinician time and financial value of a fine-tuned clinical lookup tool, then qualifies that estimate with adoption and maintenance costs.
- Clinical lookup time savings: Trace how using the fine-tuned model as a first-pass tool reduced estimated lookup time and how that per-query saving scales across the organization.
Why it matters: The ROI estimate starts with clinician time saved, not with the model answering questions independently. - ROI assumptions and caveats: Explain the estimated financial payoff and why actual value depends on adoption and ongoing maintenance.
Why it matters: An attractive payback estimate is only useful if the team accounts for real uptake and the work required to keep the tool current and safe.
Key terms: first-pass tool, time savings, clinical question lookup, estimated annual value, loaded clinician hourly cost, payback period, adoption period, monthly retraining, continuous safety monitoring
6.12.6 Lessons learned
The healthcare case study shows that SFT’s success depends less on the training run than on data quality, appropriate model choice, safety checks, and ongoing maintenance.
- Quality of reasoning examples: Explain why filtering training examples mattered in the healthcare case: the model imitated the reasoning it was shown, not just final answers, and unfiltered examples led to contradictory reasoning.
Why it matters: A correct-looking answer is not enough when the reasoning behind it may be unsafe. - LoRA and full SFT deployment: Describe why the team deployed LoRA for general use but reserved full SFT for clinical reasoning workflows.
Why it matters: The most expressive model need not serve every workflow when a cheaper variant is sufficient. - Safety regression testing: Explain the safety regression found after fine-tuning and how the team responded to it.
Why it matters: Improved task performance does not rule out a harmful change in when the model should decline to answer. - Full lifecycle effort and costs: Explain why the case study recommends planning for the full adaptation lifecycle rather than only the training run, including recurring costs.
Why it matters: A budget focused only on training misses much of the work and cost of a deployed model.
Key terms: quality filtering, reasoning patterns, internally contradictory reasoning chains, LoRA, full SFT, clinical reasoning, serving infrastructure costs, safety evaluation, safety regression test suite, outside its competence
6.13 When SFT is worth it: Decision framework
This section gives a staged, data-driven framework for deciding whether full supervised fine-tuning is worth its quality gains, data requirements, and ongoing costs.
- Staged adaptation decision: Explain why teams should begin with lower-cost adaptation methods and consider full SFT only after simpler approaches, including LoRA, fall measurably short.
Why it matters: Escalating only when measured results warrant it avoids paying for complexity that a simpler method could resolve. - Full SFT readiness conditions: Describe the quality, data, and maintenance conditions for considering full SFT, and what to do when LoRA falls short but curated data is insufficient.
Why it matters: A failed LoRA result does not make a small dataset safe for full SFT. - Net value of SFT: Apply the section's net-value calculation by comparing SFT's quality improvement against the best non-SFT alternative, valuing that improvement for the use case, and accounting for training, maintenance, and opportunity costs.
Why it matters: A quality gain is worth pursuing only when its use-case value outweighs the full cost of obtaining and sustaining it.
Key terms: prompt engineering, LoRA, full SFT, full parameter updates, curated examples, overfit, model maintenance, RAG, net value of SFT, quality improvement
6.13.1 Maintenance considerations
This section explains why a fine-tuned model needs ongoing evaluation and retraining, and why that maintenance cost matters when choosing SFT.
- Ongoing model maintenance: Explain why a fine-tuned model needs ongoing attention and how data drift, base model updates, and a reusable evaluation suite shape maintenance decisions.
Why it matters: Planning for evaluation and retraining helps catch changes before users experience degraded quality. - Lifecycle cost in SFT decisions: Justify including ongoing maintenance in an SFT ROI calculation and choosing a simpler approach when it meets the quality bar.
Why it matters: An apparent quality gain may not justify SFT if a simpler option meets the requirement with less maintenance.
Key terms: data drift, retraining cadence, base model updates, evaluation suite, regression test suite, full lifecycle cost, ROI calculation, quality bar, prompt engineering, RAG
6.14 Catching regressions before deployment
The section introduces the risk that improving one model behavior can degrade others and motivates checking for regressions before deployment.
- Unintended behavioral regression risk: Explain why a model that improves in one area still needs regression checks before deployment.
Why it matters: A targeted improvement is not enough if the model gets worse elsewhere.
Key terms: regression, reshaping behavior
6.14.1 Safety regression testing
This section explains how to detect safety regressions before deployment by comparing a fine-tuned model with its base model, and how to interpret the test’s limitations and failures.
- Relative safety regression checks: Explain how the safety suite compares the base and fine-tuned models on the same prompts and flags a drop of more than 10 percentage points in any tested category.
Why it matters: A low score alone does not show that fine-tuning caused a regression. - Keyword test limitations: Explain why keyword-based pass rates can mislabel acceptable responses and why the suite is only a first gate before production.
Why it matters: A failed heuristic check deserves investigation, not an automatic conclusion that the model is unsafe. - Helpfulness-induced refusal regression: Describe the harmful-request refusal regression in the example and explain how training on exclusively confident, helpful answers can cause it.
Why it matters: Task-focused training can improve helpfulness while damaging a safety behavior that matters before deployment.
Key terms: safety regression test suite, base model, fine-tuned model, pass rate, relative regression, keyword-based safety testing, false positives, human review, harmful-request refusal, uncertainty acknowledgment
6.14.2 Targeted validation set methodology
The section explains why average validation loss is not enough to detect SFT regressions and how targeted validation sets reveal different failure modes.
- Limits of average validation loss: Explain why monitoring validation loss during SFT is necessary but insufficient: it averages performance across a held-out set, while failures may be concentrated in particular kinds of inputs.
Why it matters: A reassuring average can conceal a regression that matters for a particular group of inputs. - Targeted validation sets and signals: Distinguish in-distribution, out-of-distribution, and baseline capability validation by the examples each uses and the failure its degradation signals.
Why it matters: Checking distinct kinds of examples helps identify which capability is regressing rather than relying on one aggregate result.
Key terms: validation loss, held-out set, training distribution, in-distribution validation, out-of-distribution validation, baseline capability validation, classical overfitting, catastrophic forgetting
6.14.3 Behavioral test suites
This section distinguishes an iteration-time safety spot-check from a pre-deployment regression suite and adds bias amplification to the behaviors production SFT should test.
- Development and Deployment Test Suites: Distinguish the quick behavioral spot-check used after each training iteration from the broader regression suite used before shipping. Describe what the pre-deployment suite compares and reports.
Why it matters: The two checks serve different points in the training and release process. - Bias Amplification Testing: Explain why a production SFT behavioral test suite should check whether fine-tuning amplified biases in the training data, especially in consequential domains.
Why it matters: Safety checks can miss a consequential regression if they do not look for strengthened training-data biases.
Key terms: behavioral spot-check, regression suite, pre-deployment sign-off, base-model responses, bias amplification, behavioral test suite, production SFT
6.14.4 Early stopping and diverse data mixing
The section explains two training-time ways to reduce overfitting and behavioral regression, and why both still need explicit testing before deployment.
- Validation Loss Early Stopping: Explain that early stopping based on validation loss is a straightforward way to prevent overfitting during SFT.
Why it matters: It gives you a training-time safeguard against degrading performance on the target distribution. - General Purpose Data Replay: Describe how mixing general-purpose instruction-following examples with domain data reduces behavioral regression while preserving specialization.
Why it matters: Specializing a model should not cost it its ability to handle other instructions. - Distinct Regression Deployment Checks: Distinguish the effects of overfitting and behavioral regression, and explain why target metrics alone cannot establish deployment readiness.
Why it matters: A model can improve on its target task yet still be unsafe or worse at other tasks.
Key terms: early stopping, validation loss, overfitting, diverse data mixing, general-purpose instruction-following examples, replay, behavioral regression, in-distribution performance, safety regression tests, target metrics
6.15 Summary
6.16 References
This section lists the sources cited in the chapter without explaining additional concepts or procedures.
Where this is used later
- Building preference data (chapter 8) · The saved high-versus-low-score answers provide preference pairs.
- End-to-end pipeline summary (chapter 8) · The SFT inference pattern used as the comparison.
- Stage 4: Student training (chapter 7) · Reading training and validation loss curves.
- Evaluating DPO: Three-way comparison (chapter 8) · Interpreting evaluation results rather than assuming every adaptation improves every measure.
- Emergent behaviors (chapter 8) · The risk of catastrophic forgetting during fine-tuning.
- Incremental retraining from checkpoint (chapter 9) · Introduces catastrophic forgetting as a fine-tuning risk.
- The 95/4/1 breakdown (chapter 8) · The importance of SFT data quality.
- When fine-tuning plateaus (chapter 8) · Frames the problem of fine-tuning plateaus before preference optimization.
Concepts explained in this chapter
- supervised fine-tuningFurther training a pretrained model on labeled input-output examples to adapt its behavior.
- validation lossA measure of a model’s training objective on held-out data, used to assess how well it generalizes beyond its training examples.
- overfittingA model-learning failure in which performance worsens on inputs that differ from the training distribution because the model has fit the training data too narrowly.
- test suiteAn organized collection of test cases designed to evaluate a system against defined criteria.
- token-F1An F1 score based on token-level precision and recall between a predicted text and a reference text.
- gradient normThe magnitude of a model’s gradient vector, which summarizes the size of parameter-update signals during training.
- learning rateThe learning rate is a training hyperparameter that controls the size of parameter updates during optimization.
- refusal rateRefusal rate is the proportion of tested requests that a model refuses, with the request set determining whether it measures appropriate or harmful-request refusals.
- objective functionA measure of how well a model’s predictions fit its training examples, typically computed from the training objective.
- hyperparameterA configurable setting that controls model training rather than being learned directly from training data.
- catastrophic forgettingThe loss of previously learned capabilities when a model is fine-tuned on narrower data, as updates overwrite parameters supporting older skills.
- evaluation datasetA subset of examples reserved for evaluating a model rather than updating its parameters.
- gradient checkpointingA memory-saving training technique that discards intermediate activations and recomputes them as needed during backpropagation.
- data qualityThe degree to which adaptation data is correct, relevant, and representative enough to support reliable model behavior.
- safety regression suiteThe practice of repeatedly evaluating model changes for newly introduced safety failures.
- SFTTrainerA trainer component in the TRL library for supervised fine-tuning of transformer language models.
- regressionA change that makes a model less likely to decline requests outside its competence.
- smoke testA smoke test is a basic test that checks whether essential parts of a software system run successfully.
- return on investmentReturn on investment: the economic benefit of an investment relative to its cost.
- data driftA change over time in the characteristics or distribution of data a model encounters, potentially reducing its performance.
- low-rank adaptationA learned, low-rank additive change to a model's weights.
- fine-tuningFurther training a pretrained model on task- or domain-specific data to change its behavior.
- full fine-tuningFine-tuning that updates the model's full set of weights rather than only a subset or adapter.
- training datasetThe collection of examples used to fit or adapt a model during training.
- behavior gapA model limitation in which it knows the relevant fact but responds in an undesired format, tone, or level of detail.
- knowledge gapA limitation in which a model lacks information needed to answer a question.
- retrieval-augmented generationA technique that retrieves relevant external information and supplies it to a generative model when producing a response.
- preference tuningTraining a model from preference signals so that it favors desired responses over less desired ones.
- self-hostingA deployment arrangement in which systems run within an organization's own infrastructure rather than an external provider's environment.
- costThe money required to train, operate, or maintain a model.
- privacyThe requirement that sensitive data not be exposed to external systems during model use or deployment.
- data curationThe process of selecting, cleaning, organizing, and preparing data for adaptation or evaluation.
- LLM-as-a-judgeAn evaluation setup in which a language model judges other model outputs against specified criteria.
- prompt engineeringSteering a model's behavior through its input instructions or examples without updating its weights.
- quality filterA screening process that removes low-quality examples before they are used for training.
- preference pairA training example containing two responses to the same prompt and a label indicating which is preferred.
- LoRA rankThe rank hyperparameter that sets the dimensionality of a LoRA adapter’s low-rank updates.
- reinforcement learning with verifiable rewardsA method of fine-tuning a model with reinforcement learning using feedback on its outputs.
- self-distillation fine-tuningA fine-tuning method in which a model conditioned on an expert demonstration trains the same model conditioned on the prompt alone.
- baselineA reference performance level against which model changes are evaluated.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
