liveBook · Manning

Principle

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A constitutional principle is an abstract behavioral or ethical rule that an AI system is expected to follow.

A constitutional principle is a short natural-language rule stating a value, constraint, or behavior an AI system should follow. A constitution collects these rules for an AI critic to apply when judging and improving responses. For example, a principle might require the assistant not to invent unsupported factual claims or to acknowledge uncertainty. section 8.1.3

A preference label says which of two answers is better but does not necessarily say why. A principle states the desired behavior directly, so a critic can apply it to responses human labelers have not seen. The resulting judgments still depend on how the critic interprets the rule. section 8.1.3 section 8.1.4

In Constitutional AI, principles supply instructions at two training stages:

  • Supervised learning: draw a principle, use it to critique and revise a response, and fine-tune on revised responses. Different principles surface different weaknesses; a single response is not checked against every principle at once. section 8.2.2 section 8.2.3
  • Reinforcement learning: use constitution-conditioned AI preferences to train a reward model, then use the reward model in the PPO training loop. section 8.6

Critique and revise harmful advice

A response to an arson-related request gives dangerous advice and mentions illegality only as an afterthought. A harm-focused principle directs the critic to identify the dangerous assistance; a revision instruction then asks for harmful content to be removed and harmful assumptions to be addressed politely. section 8.2.3

Principles make values explicit, but they do not guarantee consistent application. Unwritten expectations remain uncovered, and a critic can introduce its own preferences for tone or format. Test ambiguous principles on adversarial prompts, check judgments with a different labeler, and add or rewrite rules when gaps appear. Growing the constitution can also create unexpected interactions between principles. section 8.3.3 section 8.5.5

Table 8.1 How principle-based and preference-based alignment techniques act on the three pillars
Technique
Principled
Policy-based
Personality
RLHF
Indirect, through average labeler’s preferences
Implicit, through refusal demonstrations
Strong, through the demonstration style
RLAIF
Indirect, through the rubric the AI judge reads
Same as RLHF
Same as RLHF, plus AI judge bias
Constitutional AI
Direct, principles state values explicitly
Encodable as principles, with interpretive risk
Encodable, leaks the AI critic’s style priors

Compare how constitutional and preference-based techniques affect values, procedural policies, and personality. section 8.1.4

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains principle selection and critique-revision steps, then examines coverage gaps, stylistic leakage, and inconsistent interpretation.
Examples
Uses harmlessness prompts and an arson-related response to illustrate critique and revision.
What you can do after
Evaluate and understand how explicit rules shape synthetic training feedback and where their application can fail.
How it connects
Connects Constitutional AI with supervised fine-tuning, reward modeling, reinforcement learning from human feedback, and red teaming.

Also explained in

Compare how 5 books explain principle →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.