liveBook · Manning

Chapter 6: Supervised fine-tuning: Maximum expressiveness

What you will learn in chapter 6 of LLM Customization and Fine-Tuning

Introduces full supervised fine-tuning as a more expressive but costlier option when a LoRA adapter falls short of a measured quality target.

After this chapter you will be able to

What it builds on

Section by section

Introduction

Introduces full supervised fine-tuning as a more expressive but costlier option when a LoRA adapter falls short of a measured quality target.

Key terms: supervised fine-tuning (SFT), LoRA, low-rank constraint, network parameter, quality target, LoRA rank, plateaued, GPU memory, overfitting

6.1 Hands-on: SFT training workflow

Introduces a full-model supervised fine-tuning workflow for an IT support assistant and contrasts it with the earlier LoRA workflow.

Key terms: supervised fine-tuning, LoRA configuration, SFTTrainer, low-rank adapters, IT technical support assistant, IT support dataset, LoRA adapter

6.1.1 Data preparation

This section prepares supervised fine-tuning data by sourcing real IT questions and answers, balancing them with general examples, and converting them into a consistent chat format.

Key terms: Stack Exchange, reference response, preference pairs, Dolly, 80/20 mix, stratify, fixed seed, validation set, held-out test set, model-selection

6.1.2 Model loading and training configuration

This section explains how to configure full supervised fine-tuning conservatively to manage memory use, protect pretrained behavior, and catch problems before a full run.

Key terms: full SFT, LoRA, SFTConfig, optimizer states, activations, catastrophic forgetting, per_device_train_batch_size, gradient_accumulation_steps, micro-batch, effective batch size

6.1.3 Training script

This section explains how the SFT training script differs from LoRA, what full-model training costs, and how to interpret the run’s initial training signals.

Key terms: SFTTrainer, peft_config, full SFT, learning rate, full-model checkpoints, GPU memory usage, GPU time, LoRA, gradient norm, NaN

6.1.4 Inference

This section shows how to load a fully fine-tuned model and make an initial qualitative check of its responses.

Key terms: chat template, step-by-step IT support style, generic chat, full SFT, from_pretrained, output directory, adapter deltas

6.2 Monitoring training

This section explains how training loss, validation loss, and gradient norm reveal whether an SFT run is learning, overfitting, or becoming unstable.

Key terms: training loss, epoch, validation loss, generalization, overfitting, checkpoint, gradient norm, grad_norm, learning rate

6.2.1 Tools for monitoring training

This section shows how to view SFT training metrics with TensorBoard or W&B and what a healthy run looks like across four metrics.

Key terms: training loss, validation loss, gradient norm, learning rate schedule, TensorBoard, Weights & Biases, W&B, Hugging Face Trainer, report_to, logging_dir

6.2.2 Reading the loss curves

This section shows how training and validation loss curves, together with gradient-norm warnings, reveal whether an SFT run is learning, overfitting, underfitting, or unstable.

Key terms: loss curve, training loss, validation loss, training steps, epochs, healthy convergence, overfitting, epoch, underfitting, instability

6.2.3 Red flags and concrete thresholds

This section gives a primary signal for monitoring fine-tuning and cautions that its referenced thresholds are starting points, not fixed rules.

Key terms: validation loss, training loss, actionable thresholds, 4B to 7B models, 400 to 1,000 examples

6.3 Evaluating the fine-tuned model

This section explains why fine-tuning requires evaluation rather than an assumption that the model improved.

Key terms: fine-tuning, rigorous evaluation, memorize the training data

6.3.1 Evaluation strategy

This section explains how to evaluate an SFT model using complementary metrics, human review, and a comparison with the base model.

Key terms: target behavior, prior capabilities, production use, quantitative metrics, accuracy, F1, perplexity, ROUGE-L, qualitative review, hand-picked prompts

6.3.2 Hands-on evaluation code

This section evaluates full SFT against a base model using held-out scores and behavioral tests, then shows why small Token-F1 changes can conceal meaningful changes in output format and safety.

Key terms: held-out test set, Token-F1, exact match, safety refusal rate, full checkpoint, greedy decoding, held-out test split, evaluation noise, GPU memory, category-level breakdown

6.3.3 Making decisions from evaluation results

This section shows how to turn fine-tuning evaluation results into training changes and decide whether a model is ready to ship.

Key terms: accuracy plateaus, underperforming category, upsampling, re-evaluating, fine-tuned model, base model, success thresholds, behavioral tests, regressions

6.4 When SFT delivers value

This section identifies the tasks where full-parameter SFT can deliver more value than LoRA and explains why.

Key terms: SFT, LoRA, low-rank constraint, rigid schema, domain-specific reasoning chains, output consistency

6.4.1 Realistic expectations

This section sets realistic expectations for how much full SFT improves on a well-tuned LoRA adapter and when training data makes that improvement likely.

Key terms: full SFT, well-tuned LoRA, vanilla base model, novel output formats, data quality, data volume, overfitting, SFT, LoRA

6.5 Fine-tuning refines, doesn’t inject knowledge

This section explains why standard fine-tuning is best used to change model behavior rather than supply missing facts, and how to pair it with RAG when a task needs both.

Key terms: pre-training, fine-tuning, LoRA, full SFT, knowledge gap, behavior gap, retrieval-augmented generation (RAG), risk-rating taxonomy, Self-Distillation Fine-Tuning (SDFT), standard SFT

6.6 SFT vs. LoRA: Realistic performance expectations

This section frames the decision between full SFT and LoRA as a trade-off between accuracy gains and added resource and operational costs.

Key terms: full SFT, LoRA adapter, compute, memory, operational complexity

6.6.1 The cost-performance tradeoff

This section compares the memory and deployment costs of rank-16 LoRA with full-parameter SFT and recommends trying a higher LoRA rank before switching when performance is close to the target.

Key terms: LoRA at rank 16, full SFT, optimizer states, gradient checkpointing, LoRA adapter, full SFT checkpoint, shared base model, save_total_limit, target metric, LoRA rank

6.6.2 The forgetting risk (catastrophic forgetting)

This section explains why full SFT is more vulnerable to catastrophic forgetting than LoRA, how to reduce that risk, and why the risk matters when weighing SFT’s modest accuracy advantage against its costs.

Key terms: catastrophic forgetting, full SFT, LoRA, general-purpose skills, safety behaviors, learning rate, early stopping, validation loss, general-purpose examples, rehearsal

6.7 Data requirements

This section explains how data quality, task complexity, preparation, and assistant-turn supervision shape SFT data requirements.

Key terms: SFT, task complexity, data quality, minimum viable, LoRA, natural regularization, representative, diverse examples, chat-structured JSONL, personally identifiable information, curation

6.7.1 Quality over quantity

This section explains why full SFT benefits more from carefully curated, representative examples than from simply collecting more data, and gives practical guidance for assembling and splitting a dataset.

Key terms: correct and complete assistant responses, consistent formatting and conventions, contradictory training signal, diverse inputs within the target domain, representative difficulty distribution, edge cases, training/validation set, stratified by task type, held out, full SFT

6.8 Common pitfalls and how to avoid them

This section explains why full SFT increases both learning capacity and failure risk, then gives preventive habits for training runs.

Key terms: Full SFT, forget, overfit, smoke test, safety regression check, early stopping, low learning rate, few epochs, weight decay

6.8.1 Avoiding catastrophic forgetting

This section explains how full SFT can cause catastrophic forgetting and presents practical ways to reduce and check that risk.

Key terms: catastrophic forgetting, full SFT, LoRA, capability loss, safety regression, learning rate, epochs, general-purpose examples, rehearsal mechanism, golden set

6.8.2 Overfitting

This section explains how to recognize overfitting in supervised fine-tuning and reduce it through early stopping, regularization, more diverse data, or lower training capacity.

Key terms: overfitting, training loss, validation loss, transferable patterns, early stopping, best checkpoint, regularization, weight decay, dropout, attention layers

6.8.3 Distribution mismatch

This section explains why supervised fine-tuning can fail when curated training examples differ from production inputs, and how to detect and reduce that mismatch.

Key terms: distribution mismatch, training distribution, production distribution, validation-set metrics, production traffic, adversarial and edge-case examples, held-out set, A/B tests, task-specific metrics

6.8.4 Hyperparameter misconfiguration

This section explains how a silently misconfigured max_length can damage SFT outputs and gives three checks to catch configuration problems before a full training run.

Key terms: max_length, truncation, loss, validation metrics, chat template, 95th percentile, hyperparameters, smoke test, sample generations

6.9 Hyperparameter iteration strategy

This section gives a diagnosis-first, one-variable-at-a-time strategy for tuning SFT hyperparameters and deciding when to stop tuning and improve the data instead.

Key terms: underfitting, overfitting, loss-curve patterns, learning rate, epochs, effective batch size, gradient_accumulation_steps, warmup, weight decay, diminishing returns

6.10 Transitioning from SFT to preference optimization (DPO and RLHF)

This section explains when to move beyond SFT and how ranked responses provide the quality signal that SFT lacks.

Key terms: SFT, single target, quality distinctions, quality bar, preference optimization, DPO, GRPO, RLHF, pairs

6.10.1 The two-stage pattern

Explains why supervised fine-tuning (SFT) establishes a baseline before preference optimization refines response quality.

Key terms: SFT, cold start, preference optimization, preference signal, GRPO

6.10.2 Preparing your SFT checkpoint for preference optimization

The section explains what to preserve from SFT, how to record its evaluation baseline, and when to deploy the checkpoint or use it for preference optimization.

Key terms: full model checkpoint, tokenizer files, SFTConfig, evaluation report JSON, DPO baseline, overall Token-F1, per-category scores, safety refusal rate, eval report JSON, SFT checkpoint

6.11 Platform-specific implementation

This section frames enterprise SFT platform selection as a trade-off between operational simplicity and fine-grained control, using three representative starting points.

Key terms: managed API, cloud ML platform, self-hosted open-source approach, operational simplicity, fine-grained control

6.11.1 OpenAI and Azure OpenAI

This section explains the managed SFT workflow on OpenAI and Azure OpenAI, and the platform and model-availability constraints that affect planning.

Key terms: SFT, JSONL, chat-completion format, checkpointing, Azure OpenAI, GPT-4.1 family, supervised fine-tuning, model list, epoch count, batch size

6.11.2 Google Vertex AI

This subsection distinguishes Vertex AI’s managed Gemini tuning from its custom training jobs and notes that Vertex handles LoRA configuration internally.

Key terms: managed Gemini tuning pathway, custom training job pathway, open-weight models, LoRA, adapter parameters

6.11.3 Hugging Face self-hosted

This subsection outlines the control-versus-management tradeoff of self-hosted SFT and the infrastructure, distributed-training, and tracking choices it requires.

Key terms: self-hosted pathway, infrastructure management, SFTTrainer, Cloud VMs with GPUs, on-premises GPU servers, managed platforms, DeepSpeed, FSDP, TrainingArguments

6.11.4 Cost and privacy considerations

This section compares SFT platform costs and privacy guarantees, then gives a rule for choosing between managed and self-hosted training.

Key terms: managed APIs, per training token, self-hosted infrastructure, per GPU-hour, TCO, self-hosted training, tenant-level isolation, enterprise agreement, HIPAA BAA, SOC 2

6.12 Real case study: Healthcare clinical Q&A

Introduces an illustrative clinical Q&A fine-tuning case study and clarifies that its figures represent published research outcomes, not one deployment.

Key terms: illustrative composite, medical question-answer pairs, clinical decision support, lookup and summarize

6.12.1 Dataset and preparation

This section describes how the clinical Q&A case study selected, formatted, cleaned, and split its data for supervised fine-tuning and evaluation.

Key terms: FreedomIntelligence/medical-o1-reasoning-SFT, clinical reasoning, drug interactions, patient communication, multi-turn conversation, structured reasoning, deduplication, length filtering, stratified sample, validation set

6.12.2 Training configuration

The healthcare case study compares LoRA with full SFT and specifies the configuration used for the full SFT run.

Key terms: LoRA, full SFT, additional capacity, Qwen3-4B-Instruct, training examples, epochs, gradient accumulation, effective batch size

6.12.3 Results

This section explains how the clinical Q&A model was evaluated and why supervised fine-tuning improved some categories more than others.

Key terms: rubric-based evaluation, expert-verified reference answers, drug interactions, clinical reasoning, general medical knowledge, SFT, patient communication, plain-language explanations, clinical jargon, appropriate register

6.12.4 Cost breakdown

The section explains why staff time, rather than GPU compute, dominates the cost of fine-tuning the case-study model.

Key terms: compute spend, staff time, data curation, evaluation

6.12.5 ROI calculation

This section estimates the clinician time and financial value of a fine-tuned clinical lookup tool, then qualifies that estimate with adoption and maintenance costs.

Key terms: first-pass tool, time savings, clinical question lookup, estimated annual value, loaded clinician hourly cost, payback period, adoption period, monthly retraining, continuous safety monitoring

6.12.6 Lessons learned

The healthcare case study shows that SFT’s success depends less on the training run than on data quality, appropriate model choice, safety checks, and ongoing maintenance.

Key terms: quality filtering, reasoning patterns, internally contradictory reasoning chains, LoRA, full SFT, clinical reasoning, serving infrastructure costs, safety evaluation, safety regression test suite, outside its competence

6.13 When SFT is worth it: Decision framework

This section gives a staged, data-driven framework for deciding whether full supervised fine-tuning is worth its quality gains, data requirements, and ongoing costs.

Key terms: prompt engineering, LoRA, full SFT, full parameter updates, curated examples, overfit, model maintenance, RAG, net value of SFT, quality improvement

6.13.1 Maintenance considerations

This section explains why a fine-tuned model needs ongoing evaluation and retraining, and why that maintenance cost matters when choosing SFT.

Key terms: data drift, retraining cadence, base model updates, evaluation suite, regression test suite, full lifecycle cost, ROI calculation, quality bar, prompt engineering, RAG

6.14 Catching regressions before deployment

The section introduces the risk that improving one model behavior can degrade others and motivates checking for regressions before deployment.

Key terms: regression, reshaping behavior

6.14.1 Safety regression testing

This section explains how to detect safety regressions before deployment by comparing a fine-tuned model with its base model, and how to interpret the test’s limitations and failures.

Key terms: safety regression test suite, base model, fine-tuned model, pass rate, relative regression, keyword-based safety testing, false positives, human review, harmful-request refusal, uncertainty acknowledgment

6.14.2 Targeted validation set methodology

The section explains why average validation loss is not enough to detect SFT regressions and how targeted validation sets reveal different failure modes.

Key terms: validation loss, held-out set, training distribution, in-distribution validation, out-of-distribution validation, baseline capability validation, classical overfitting, catastrophic forgetting

6.14.3 Behavioral test suites

This section distinguishes an iteration-time safety spot-check from a pre-deployment regression suite and adds bias amplification to the behaviors production SFT should test.

Key terms: behavioral spot-check, regression suite, pre-deployment sign-off, base-model responses, bias amplification, behavioral test suite, production SFT

6.14.4 Early stopping and diverse data mixing

The section explains two training-time ways to reduce overfitting and behavioral regression, and why both still need explicit testing before deployment.

Key terms: early stopping, validation loss, overfitting, diverse data mixing, general-purpose instruction-following examples, replay, behavioral regression, in-distribution performance, safety regression tests, target metrics

6.15 Summary

6.16 References

This section lists the sources cited in the chapter without explaining additional concepts or procedures.

Where this is used later

Concepts explained in this chapter

← Chapter 5: Parameter-Efficient Fine-Tuning: LoRA and QLoRA Chapter 7: Knowledge distillation: Capturing frontier model intelligence →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.