liveBook · Manning

Alignment training

As explained in LLM Customization and Fine-Tuning

Steering a model's behavior toward human or organizational preferences and constraints.

A model can perform its task well yet fail to refuse harmful requests or follow a required tone. Alignment training targets behaviors or constraints that the base model resists. It can use preference pairs to steer behavior, but it does not replace evaluation of the resulting model. section 1.7 section 7.9

Alignment sits toward the weight-update end of the adaptation continuum, where engineering cost, GPU requirements, and timelines increase. Start by measuring a baseline: prompting or retrieval may be sufficient for the task. If a specific behavior or tone remains resistant, alignment is a candidate; frequently changing knowledge is instead a reason to consider retrieval. section 1.6.2 section 1.7

Prepare curated task examples and a held-out evaluation set before choosing training. The same instruction-response pairs can seed several adaptation techniques, while preference pairs are an alignment-specific addition. Privacy and data residency determine where the model runs, rather than which adaptation technique to use. section 1.7

Do not assume that a distilled student inherits its teacher’s safety alignment. In black-box distillation, benign outputs alone do not teach the student when or how to refuse. Include harmful prompts paired with teacher refusals in the distillation data, then retest the student. Safety-aware spot checks are also needed: fluent, coherent teacher outputs can still be harmful. section 7.8 section 7.8.1

Safety regression in medical distillation

In a reported medical-domain study, distilled models produced unsafe content on 86% of red-team prompts even though the teacher refused all of them. Task performance was preserved, but safety alignment was lost. Test each student independently for safety regressions before deployment. section 7.8.1 section 7.9

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains safety loss during distillation and its mitigation, without detailing a preference-optimization procedure.
Examples
Uses a medical-domain distillation study where students produced unsafe content on 86% of red-team prompts refused by the teacher.
What it assumes
Uses a measured baseline, curated task examples, and a held-out evaluation set before selecting a training technique.
What you can do after
Evaluate and understand when alignment may address a resistant behavior, and test distilled students for safety regressions.
How it connects
Connects alignment to preference optimization, distillation, supervised fine-tuning, retrieval-augmented generation, and safety evaluation.

Also explained in

Compare how 9 books explain alignment training →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.