A constitutional principle is a short natural-language rule stating a value, constraint, or behavior an AI system should follow. A constitution collects these rules for an AI critic to apply when judging and improving responses. For example, a principle might require the assistant not to invent unsupported factual claims or to acknowledge uncertainty. section 8.1.3
A preference label says which of two answers is better but does not necessarily say why. A principle states the desired behavior directly, so a critic can apply it to responses human labelers have not seen. The resulting judgments still depend on how the critic interprets the rule. section 8.1.3 section 8.1.4
In Constitutional AI, principles supply instructions at two training stages:
- Supervised learning: draw a principle, use it to critique and revise a response, and fine-tune on revised responses. Different principles surface different weaknesses; a single response is not checked against every principle at once. section 8.2.2 section 8.2.3
- Reinforcement learning: use constitution-conditioned AI preferences to train a reward model, then use the reward model in the PPO training loop. section 8.6
Critique and revise harmful advice
A response to an arson-related request gives dangerous advice and mentions illegality only as an afterthought. A harm-focused principle directs the critic to identify the dangerous assistance; a revision instruction then asks for harmful content to be removed and harmful assumptions to be addressed politely. section 8.2.3
Principles make values explicit, but they do not guarantee consistent application. Unwritten expectations remain uncovered, and a critic can introduce its own preferences for tone or format. Test ambiguous principles on adversarial prompts, check judgments with a different labeler, and add or rewrite rules when gaps appear. Growing the constitution can also create unexpected interactions between principles. section 8.3.3 section 8.5.5
Table 8.1 How principle-based and preference-based alignment techniques act on the three pillars
|
Technique
|
Principled
|
Policy-based
|
Personality
|
|
RLHF
|
Indirect, through average labeler’s preferences
|
Implicit, through refusal demonstrations
|
Strong, through the demonstration style
|
|
RLAIF
|
Indirect, through the rubric the AI judge reads
|
Same as RLHF
|
Same as RLHF, plus AI judge bias
|
|
Constitutional AI
|
Direct, principles state values explicitly
|
Encodable as principles, with interpretive risk
|
Encodable, leaks the AI critic’s style priors
|
Compare how constitutional and preference-based techniques affect values, procedural policies, and personality. section 8.1.4




