A helpful answer can still cause harm, mislead a user, or conflict with a policy. Principled alignment shapes model behavior around foundational values such as honesty, helpfulness, and harmlessness. The engineering challenge is turning those abstract values into behavior that can be trained and evaluated. section 1.2.1 section 8.0
Principled alignment is one of three interacting pillars. Policy alignment enforces specific rules and procedures; personality alignment shapes tone and communication style. A customer service assistant, for example, must balance being helpful with following company procedures and maintaining a consistent voice. section 1.2.1 section 8.1.4
A preference can combine several judgments
An annotator compares two financial summaries. One cites the source and uses a neutral tone; the other invents a geographic explanation and sounds more confident. Preferring the first rewards honesty, adherence to citation instructions, and professional style at once. The single preference does not reveal how much each dimension contributed. section 7.1.2
Reinforcement learning from human feedback (RLHF) can learn values indirectly from aggregated preferences, but its blended signal makes individual pillars hard to control. Demonstrations alone can teach a model to imitate helpful answers without reliably establishing a general preference for helpfulness. section 7.1.2 section 7.5.2
Constitutional AI makes values explicit in a set of principles. An AI critic uses them to critique and revise responses and choose between alternatives; those outputs become training data. For example, a directive to choose the more honest response places honesty directly in the judgment process. This is an operational use of the principle, not evidence that the critic philosophically understands honesty. section 8.0 section 8.1.4
Turn a value into a training signal
For a financial research assistant, write a principle not to invent facts absent from the source. An AI critic can use it to label variants of single-quarter questions, multi-quarter trends, and cross-company comparisons. Check the critic carefully: systematic errors in its judgments can be amplified by training. section 8.2.5
Table 8.1 How principle-based and preference-based alignment techniques act on the three pillars
|
Technique
|
Principled
|
Policy-based
|
Personality
|
|
RLHF
|
Indirect, through average labeler’s preferences
|
Implicit, through refusal demonstrations
|
Strong, through the demonstration style
|
|
RLAIF
|
Indirect, through the rubric the AI judge reads
|
Same as RLHF
|
Same as RLHF, plus AI judge bias
|
|
Constitutional AI
|
Direct, principles state values explicitly
|
Encodable as principles, with interpretive risk
|
Encodable, leaks the AI critic’s style priors
|
Compare how preference-based and principle-based methods act on the three alignment pillars, including their interpretive risks. section 8.1.4
Explicit principles do not eliminate judgment calls. A critic may interpret procedural rules inconsistently on new inputs, and its own style preferences can influence personality. Evaluate outcomes by pillar rather than treating an aggregate improvement as proof that every dimension improved. section 8.1.4 section 7.1.2
