Alignment training

As explained in Generative AI in Action

Alignment is the process of shaping an AI model's behavior to conform to intended goals, values, or constraints.

Alignment training shapes an AI model toward intended goals, human values, and safety constraints. Common approaches include fine-tuning and reinforcement learning from human feedback (RLHF), but alignment does not guarantee harmless or human-identical behavior. section 13.3.3 section 12.4.3

Apply alignment as part of a broader production safety strategy:

• Inspect model cards and transparency notes to understand safety techniques and known problems. section 13.3.3
• Red-team different model versions and measure potential harms rather than assuming alignment is sufficient. section 13.3.3
• Layer model-level alignment with safety systems, user review, application controls, and published policies. section 13.3.3

Evaluating moral and causal alignment

The MoCa framework compares LLM judgments about causal and moral scenarios with human participants’ judgments. It uses AuROC as its main alignment metric and accuracy as a secondary measure. Newer, larger models such as GPT-4 and Claude score closer to human judgments than smaller models such as GPT-3, although no model perfectly matches human moral systems. section 12.4.3

Figure 13.11 Harms mitigation layers

Use the harms-mitigation layers to position alignment training alongside safety monitoring, application design, user interventions, and governance. section 13.3.3

How Generative AI in Action covers it

How deep it goes

Identifies fine-tuning and RLHF, then emphasizes testing and layered mitigations without detailing training procedures.

Examples

Uses GPT-4, Claude, GPT-3, Llama 2, RLHF, MoCa scenarios, AuROC, and human moral judgments.

What you can do after

Evaluate alignment with human judgments and design layered safeguards around aligned models.

How it connects

Connects alignment with reinforcement learning from human feedback, fine-tuning, red-teaming, safety systems, and model evaluation.

liveBookmanning.com

Read Generative AI in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  WordPress Power Toolkit: Harness AI to build next-level websitesThe alignment problem is the challenge of ensuring that an AI system’s behavior remains consistent with human values and goals.
• 

  liveBookmanning.com
  How Large Language Models WorkThe alignment problem is the challenge of ensuring that an artificial system's goals and methods remain consistent with the goals and values intended by humans.
• 

  liveBookmanning.com
  Causal AIAI alignment is the task of making an artificial intelligence system's behavior consistent with human goals and preferences.
• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionAlignment is the property of an artificial-intelligence system whose behavior is consistent with intended human goals, values, and constraints.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackValue alignment is the conformity of an intelligent system's objectives and behavior to the values held by the people or population it is intended to serve.

Compare how 6 books explain alignment training →

Related concepts

• distillationcontrasts with · Knowledge distillation is a training technique in which a student model learns to reproduce or approximate the outputs or behavior of a teacher model.
• area under the curvemeasured by · The area under a curve is the integral of its values over a specified domain and, in classification evaluation, commonly summarizes performance across decision thresholds.
• evaluationcontrasts with · Evaluation is the systematic assessment of a system, result, or approach against relevant criteria, evidence, or objectives.
• behaviormeasured by · Behavior is the externally observable actions, responses, outputs, or effects of a system or object under specified conditions or interactions.

This explainer was generated by Manning's knowledge graph from the text of Generative AI in Action. The section links point to the parts of the book it draws on.