Anthropic developed a technique for training a model to be more harmless that does not depend on humans labeling which of two harmful responses is worse, since asking human labelers to compare harmful outputs to each other raises its own problems. Instead, the model is trained against a written list of principles (a 'constitution'). How does Bai et al.'s Constitutional AI method actually use that constitution across its two training phases?
- In a first, supervised phase, the model critiques and revises its own sampled responses against the constitution's principles and is fine-tuned on the revised outputs; in a second, reinforcement-learning phase, the model itself judges which of two sampled responses better satisfies the constitution to build an AI-generated preference dataset, which trains a preference model used as the reward signal
- The model is fine-tuned once, in a single supervised phase, purely on human-labeled comparisons between harmful responses, and the written constitution is used only as documentation describing the labeling instructions given to the human annotators
- The constitution is applied only as a static output filter at inference time, blocking any response that matches a principle in the list, with no fine-tuning of the underlying model's weights involved at any stage
- The constitution's principles are converted into a fixed set of rules and injected directly into the model's weights through a rule-based, gradient-free editing procedure, bypassing both supervised fine-tuning and reinforcement learning entirely
Why A? And why not the others?
Correct answer: A. In a first, supervised phase, the model critiques and revises its own sampled responses against the constitution's principles and is fine-tuned on the revised outputs; in a second, reinforcement-learning phase, the model itself judges which of two sampled responses better satisfies the constitution to build an AI-generated preference dataset, which trains a preference model used as the reward signal
Constitutional AI runs in two stages: a supervised-learning stage where the model generates a response, then critiques and revises that response against a written set of principles, and is fine-tuned directly on those self-revised outputs; followed by a reinforcement-learning stage where the model -- rather than a human -- compares pairs of sampled responses against the same constitution to build a dataset of AI-generated preference judgments, which is used to train a preference model that then supplies the reward signal for reinforcement learning, a process the paper calls reinforcement learning from AI feedback (RLAIF). The option describing a single supervised phase trained purely on human-labeled harmful-response comparisons is wrong because it omits the entire point of the method, which is to replace human labeling of harmful comparisons with the model's own judgments. The option describing a static inference-time filter is wrong because both phases involve actually updating the model's weights through training, not filtering outputs after the fact. The option describing a gradient-free rule-injection procedure is wrong because both phases described in the paper use ordinary gradient-based supervised fine-tuning and reinforcement learning, not direct weight editing.
Source: Bai et al., 'Constitutional AI: Harmlessness from AI Feedback' (arXiv:2212.08073, 2022)