Ouyang et al. (2022), "Training Language Models to Follow Instructions with Human Feedback" (the InstructGPT paper), describes a three-stage pipeline for aligning a pretrained language model with human preferences. What are the three stages, in order?
- Pretrain on a labeled classification dataset, then deploy directly without further tuning, since pretraining alone is assumed to capture human preferences
- Supervised fine-tuning on human-written demonstrations, followed by training a reward model on human rankings of sampled model outputs, followed by reinforcement learning (PPO) that optimizes the policy against that reward model
- Train a reward model on raw internet text first, then use it to filter the pretraining corpus before any supervised fine-tuning occurs
- Fine-tune the model using only reinforcement learning against a fixed rule-based reward function, without using any human-labeled data at any stage
Why B? And why not the others?
Correct answer: B. Supervised fine-tuning on human-written demonstrations, followed by training a reward model on human rankings of sampled model outputs, followed by reinforcement learning (PPO) that optimizes the policy against that reward model
Ouyang et al. describe reinforcement learning from human feedback (RLHF) as a three-stage process applied on top of an already-pretrained language model. First, the model is supervised fine-tuned on a dataset of human-written demonstrations of desired outputs. Second, human labelers rank multiple model outputs for the same prompts by quality, and this comparison data is used to train a separate reward model that predicts which output humans would prefer. Third, the fine-tuned model is further optimized using Proximal Policy Optimization (PPO), treating the reward model's score as the reward signal to maximize. The option describing pretraining alone followed by direct deployment is wrong because it skips both the demonstration-based fine-tuning and the preference-based reward modeling entirely. The option starting with a reward model trained on raw internet text is wrong because the reward model is trained on human comparison rankings of model outputs, not used to pre-filter the pretraining corpus. The option describing pure rule-based reinforcement learning with no human data is wrong because both the demonstration data and the comparison rankings are human-generated and essential to the pipeline.
Source: Ouyang et al., "Training Language Models to Follow Instructions with Human Feedback" (2022), arXiv:2203.02155, Section 3