8 Alignment through principles: Constitutional AI
This chapter covers
- Reinforcement learning from AI feedback guided by written principles
- The helpfulness-honesty-harmlessness trade-off
- Designing a constitution, from Anthropic’s principle list to OpenAI’s Model Spec
- Teaching a model the spec through midtraining
Reinforcement learning from human feedback has a scaling problem. Human labelers are the bottleneck for reward modeling. InstructGPT used about 13,000 SFT prompts, 33,000 reward-model prompts, and 31,000 PPO prompts. By post-training standards, that is a small dataset. Once a team needs to update its assistant for a new product surface, policy area, or to fix misalignments that emerged in production, the cost of collecting another round of labeled comparisons starts to dominate. The hallucination examples from the last chapter are not a one-time fix. Every new earnings cycle or out-of-training distribution data introduces new transcripts, entities, and ways for financial research assistants to fabricate plausible-sounding names. RLHF can train against the failures the labelers have already seen, but the human labeler workforce cannot scale to cover every variant in a real product distribution.