How deeply does Reinforcement Learning from Human Feedback cover constitution-as-code document?

Formalizes principles, iterative revisions, final training pairs, and preference comparisons without detailing the complete downstream optimization.

Constitution-as-code document

As explained in Reinforcement Learning from Human Feedback

A constitution is a human-written set of principles specifying aspects of model behavior to assess during critique.

A constitution is a human-written set of principles that makes desired model behavior explicit and gives critique a repeatable target. Principles such as avoiding violence or remaining truthful turn broad alignment goals into checks applied to model outputs. section 12.5

A constitution supports two synthetic-data pipelines:

Instruction refinement repeatedly samples a principle, critiques the latest completion, and revises it until the resulting prompt–completion pair is suitable for fine-tuning. section 12.5
Preference generation gives a feedback model a prompt, selected principles, and competing completions so it can identify the better answer. section 12.5
The resulting synthetic preference data can enter RLHF-style optimization, commonly described in this setting as RLAIF. section 12.5

Principle-guided revision

Given a prompt x and an initial completion y⁰, the process samples principles cᵢ from the constitution, asks the model to revise its latest answer, and produces y⁰ through yⁿ. The final pair is x with yⁿ; a separate feedback model can instead compare two completions against selected principles. section 12.5

Constitutions also function as behavior specifications. Later documents can explain the reasoning and intent behind principles, helping designers clarify priorities and stakeholders compare intended behavior with observed model behavior. section 17.2

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

Constitutional AIused by · Constitutional AI is an alignment approach that guides model behavior by evaluating or shaping it according to an explicit set of principles.
evaluatorused by · An evaluator is a model or component that assesses outputs against specified criteria and produces judgments, scores, comparisons, or feedback.
principlehas part · A principle is a general rule, guideline, or value that guides decisions, behavior, or judgments.
specificationkind of · A specification is a formal or informal description of the required or intended behavior, properties, interfaces, or constraints of a system or component.
behaviorconfigures · Behavior is the externally observable actions, responses, outputs, or effects of a system or object under specified conditions or interactions.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.