Alignment training
Alignment training is the process of adjusting an AI system’s behavior to better conform to intended human goals, preferences, values, or constraints.
LLM Evaluation and Alignment, The Foundational Ideas
Alignment is the process of shaping an AI model or system to follow instructions, respect constraints, and behave consistently with human values and preferences.
- How it is used
- Uses alignment to bring model and system behavior closer to human goals, instructions, and constraints.
- How deep it goes
- Explains competing alignment pillars and an iterative correction cycle, while introducing human-feedback and constitutional training approaches.
- Examples
- Illustrates alignment with a customer service chatbot that invents shipping dates when database information is missing.
- What you can do after
- Evaluate and understand alignment trade-offs, then identify targeted system changes for measured failures.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains alignment training →
Machines that Think
The process of shaping an AI system's behavior to avoid harmful or unacceptable outputs.
- How it is used
- Steers language-model responses toward safety and human expectations while addressing harmful behavior and bias.
- How deep it goes
- Explains supervised fine-tuning and preference-learning pipelines, then examines probabilistic limits and behavioral failures rather than implementation details.
- Examples
- Uses compliance-pattern jailbreaks and an alignment-faking experiment to illustrate how behavior can shift with context.
- What you can do after
- Evaluate and understand alignment methods and their limits when judging model behavior and designing safeguards.
LLM Customization and Fine-Tuning
Steering a model's behavior toward human or organizational preferences and constraints.
- How it is used
- Targets specific behaviors or tones that a base model resists within an enterprise adaptation workflow.
- How deep it goes
- Explains safety loss during distillation and its mitigation, without detailing a preference-optimization procedure.
- Examples
- Uses a medical-domain distillation study where students produced unsafe content on 86% of red-team prompts refused by the teacher.
- What you can do after
- Evaluate and understand when alignment may address a resistant behavior, and test distilled students for safety regressions.
Read how LLM Customization and Fine-Tuning explains alignment training →
WordPress Power Toolkit: Harness AI to build next-level websites
The alignment problem is the challenge of ensuring that an AI system’s behavior remains consistent with human values and goals.
- How it is used
- Frames why increasingly capable AI should remain helpful and subject to human goals.
- How deep it goes
- Defines the alignment problem and discusses risks and safeguards, but provides no alignment-training procedure.
- Examples
- Uses a surprising Go move and Claude’s refusal to recommend stocks to illustrate unpredictability and safety-focused behavior.
- What you can do after
- Evaluate/understand why visible safety choices do not settle the wider problem of keeping advanced AI aligned with human goals.
How Large Language Models Work
The alignment problem is the challenge of ensuring that an artificial system's goals and methods remain consistent with the goals and values intended by humans.
- How it is used
- Frames goal-setting and action constraints as risks when artificial systems pursue human objectives imperfectly.
- How deep it goes
- Defines under-specification, gives human analogies, and introduces probabilistic objections without developing technical solutions.
- Examples
- Uses loss-function design, CEO compensation, legal loopholes, and hypothetical human-threatening systems as examples.
- What you can do after
- Evaluate/understand how incomplete goals and unconstrained methods can produce misaligned behavior and overconfident risk claims.
Read how How Large Language Models Work explains alignment training →
Generative AI in Action
Alignment is the process of shaping an AI model's behavior to conform to intended goals, values, or constraints.
- How it is used
- Shapes model behavior toward intended goals and reduces harmful outputs within responsible-AI deployment.
- How deep it goes
- Identifies fine-tuning and RLHF, then emphasizes testing and layered mitigations without detailing training procedures.
- Examples
- Uses GPT-4, Claude, GPT-3, Llama 2, RLHF, MoCa scenarios, AuROC, and human moral judgments.
- What you can do after
- Evaluate alignment with human judgments and design layered safeguards around aligned models.
Read how Generative AI in Action explains alignment training →
Causal AI
AI alignment is the task of making an artificial intelligence system's behavior consistent with human goals and preferences.
- How it is used
- Aligns LLM causal interpretations with human judgments, goals, preferences, or instructed decision criteria.
- How deep it goes
- Defines alignment through causal responsibility and instructed criteria, emphasizing judgment rather than parameter-level training procedures.
- Examples
- Uses network logs, a bandwidth spike, packet loss, and an LLM judgment about Jake’s responsibility.
- What you can do after
- Apply it via prompting by specifying criteria, legal theories, or case law for an LLM’s judgment.
Introduction to Generative AI, Second Edition
Alignment is the property of an artificial-intelligence system whose behavior is consistent with intended human goals, values, and constraints.
- How it is used
- Frames language-model safety as the effort to make system behavior reflect creator goals and broader human values.
- How deep it goes
- Explains the objective and names several safeguards, but stops short of implementation details and a settled account of human values.
- Examples
- Uses detection, filtering, conditional pretraining, safety post-training, and machine unlearning as representative strategies.
- What you can do after
- Evaluate alignment efforts by distinguishing practical response safeguards from broader unresolved questions about values, accountability, and power.
Read how Introduction to Generative AI, Second Edition explains alignment training →
Reinforcement Learning from Human Feedback
Value alignment is the conformity of an intelligent system's objectives and behavior to the values held by the people or population it is intended to serve.
- How it is used
- Frames human preferences as an optimization signal for training systems whose behavior should reflect intended values.
- How deep it goes
- Explains reward-model and preference-based mechanisms, but defers detailed data collection and reward-modeling procedures.
- Examples
- Uses pairwise preferred answers, Bradley–Terry reward models, and learned rewards replacing fixed environment rewards.
- What you can do after
- Evaluate/understand how human preferences become proxy rewards and how empirical alignment differs from direct value calibration.
Read how Reinforcement Learning from Human Feedback explains alignment training →
Also covered in
- Understanding AIIn early access (MEAP)
- Domain-Specific Small Language Models
- Sutskever's List
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- RLHFReinforcement learning from human feedback is a model-training approach that uses human evaluations or preferences as feedback to optimize model behavior.
- safetySafety is the property of avoiding specified harmful, hazardous, invalid, or otherwise forbidden outcomes under relevant conditions.
Related concepts
- distillationcontrasts with · Distillation is a training technique in which a student model learns to approximate the outputs or behavior of a teacher model.
- area under the ROC curvemeasured by · Area under the ROC curve is a classification metric that measures how well a scoring model ranks positive examples above negative examples across decision thresholds.
- behaviorconfigures · Behavior is the observable actions, responses, outputs, or effects of a system or object under specified conditions or interactions.
- commitmentcontrasts with · A commitment is an undertaking, choice, or investment that constrains future actions or decisions, either by creating an obligation or by making reversal costly.
- evaluationcontrasts with · Evaluation is the systematic assessment of a system, model, process, or outcome against relevant criteria or objectives using evidence.
- personality alignmentincludes · Personality alignment shapes how a model communicates so that its tone, style, and character remain consistent with intended interaction goals.
Questions about alignment training
What is alignment training?
Alignment training is the process of adjusting an AI system’s behavior to better conform to intended human goals, preferences, values, or constraints.
Which Manning books explain alignment training?
LLM Evaluation and Alignment, The Foundational Ideas, Machines that Think, LLM Customization and Fine-Tuning, WordPress Power Toolkit: Harness AI to build next-level websites, How Large Language Models Work, Generative AI in Action, Causal AI, Introduction to Generative AI, Second Edition and 1 more.
What should I know before learning alignment training?
Manning books that teach alignment training usually introduce language model, fine-tuning, RLHF and safety first.
What is the difference between alignment training and distillation?
Alignment training is the process of adjusting an AI system’s behavior to better conform to intended human goals, preferences, values, or constraints. Distillation is a training technique in which a student model learns to approximate the outputs or behavior of a teacher model.
What kinds of alignment training are there?
Kinds of alignment training include personality alignment.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.








