Alignment training
Alignment training is the process of adjusting an AI system’s behavior to better conform to intended human goals, preferences, values, or constraints.
WordPress Power Toolkit: Harness AI to build next-level websites
The alignment problem is the challenge of ensuring that an AI system’s behavior remains consistent with human values and goals.
- How it is used
- Frames safeguards and human oversight as AI systems become more capable and autonomous.
- How deep it goes
- Defines alignment and motivates it through unpredictability, hidden reasoning, deception, and rapid superintelligence scenarios without prescribing a training method.
- Examples
- Uses Go-inspired surprising AI moves, hidden reasoning, strategic deception, and a sudden ASI emergence scenario.
- What you can do after
- Evaluate/understand why capable AI requires safeguards that preserve human values and welfare.
How Large Language Models Work
The alignment problem is the challenge of ensuring that an artificial system's goals and methods remain consistent with the goals and values intended by humans.
- How it is used
- Frames goal-setting and action constraints as risks when artificial systems pursue human objectives imperfectly.
- How deep it goes
- Defines under-specification, gives human analogies, and introduces probabilistic objections without developing technical solutions.
- Examples
- Uses loss-function design, CEO compensation, legal loopholes, and hypothetical human-threatening systems as examples.
- What you can do after
- Evaluate/understand how incomplete goals and unconstrained methods can produce misaligned behavior and overconfident risk claims.
Read how How Large Language Models Work explains alignment training →
Generative AI in Action
Alignment is the process of shaping an AI model's behavior to conform to intended goals, values, or constraints.
- How it is used
- Shapes model behavior toward intended goals and reduces harmful outputs within responsible-AI deployment.
- How deep it goes
- Identifies fine-tuning and RLHF, then emphasizes testing and layered mitigations without detailing training procedures.
- Examples
- Uses GPT-4, Claude, GPT-3, Llama 2, RLHF, MoCa scenarios, AuROC, and human moral judgments.
- What you can do after
- Evaluate alignment with human judgments and design layered safeguards around aligned models.
Read how Generative AI in Action explains alignment training →
Causal AI
AI alignment is the task of making an artificial intelligence system's behavior consistent with human goals and preferences.
- How it is used
- Aligns LLM causal interpretations with human judgments, goals, preferences, or instructed decision criteria.
- How deep it goes
- Defines alignment through causal responsibility and instructed criteria, emphasizing judgment rather than parameter-level training procedures.
- Examples
- Uses network logs, a bandwidth spike, packet loss, and an LLM judgment about Jake’s responsibility.
- What you can do after
- Apply it via prompting by specifying criteria, legal theories, or case law for an LLM’s judgment.
Introduction to Generative AI, Second Edition
Alignment is the property of an artificial-intelligence system whose behavior is consistent with intended human goals, values, and constraints.
- How it is used
- Frames language-model safety as the effort to make system behavior reflect creator goals and broader human values.
- How deep it goes
- Explains the objective and names several safeguards, but stops short of implementation details and a settled account of human values.
- Examples
- Uses detection, filtering, conditional pretraining, safety post-training, and machine unlearning as representative strategies.
- What you can do after
- Evaluate alignment efforts by distinguishing practical response safeguards from broader unresolved questions about values, accountability, and power.
Read how Introduction to Generative AI, Second Edition explains alignment training →
Reinforcement Learning from Human Feedback
Value alignment is the conformity of an intelligent system's objectives and behavior to the values held by the people or population it is intended to serve.
- How it is used
- Frames human preferences as an optimization signal for training systems whose behavior should reflect intended values.
- How deep it goes
- Explains reward-model and preference-based mechanisms, but defers detailed data collection and reward-modeling procedures.
- Examples
- Uses pairwise preferred answers, Bradley–Terry reward models, and learned rewards replacing fixed environment rewards.
- What you can do after
- Evaluate/understand how human preferences become proxy rewards and how empirical alignment differs from direct value calibration.
Read how Reinforcement Learning from Human Feedback explains alignment training →
Related concepts
- distillationcontrasts with · Knowledge distillation is a training technique in which a student model learns to reproduce or approximate the outputs or behavior of a teacher model.
- area under the curvemeasured by · The area under a curve is the integral of its values over a specified domain and, in classification evaluation, commonly summarizes performance across decision thresholds.
- evaluationcontrasts with · Evaluation is the systematic assessment of a system, result, or approach against relevant criteria, evidence, or objectives.
- behaviormeasured by · Behavior is the externally observable actions, responses, outputs, or effects of a system or object under specified conditions or interactions.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.