A team wants to align a pretrained language model's outputs with human preferences -- for example, favoring helpful, well-formatted answers over unhelpful or poorly formatted ones -- in a way that goes beyond what the original next-token-prediction pretraining objective optimizes for. They collect many pairs of candidate outputs for the same prompt and have human annotators indicate which output in each pair they prefer. How does RLHF (reinforcement learning from human feedback) typically use this preference data to align the model?
- It directly edits the pretrained model's weight matrices using the preference labels as targets in an ordinary supervised next-token-prediction loss, with no reward model or reinforcement learning step involved at all
- It uses the preference pairs only to filter the pretraining corpus, removing any documents similar in style to the less-preferred outputs, before retraining the model from scratch on the filtered corpus
- It first trains a separate reward model to predict which of two outputs a human would prefer, using the collected pairwise comparisons as training data, and then uses reinforcement learning (commonly PPO) to fine-tune the language model's policy to maximize the score that reward model assigns to its outputs
- It uses the preference pairs to build a retrieval index, and at inference time retrieves the most similar previously-preferred output to return as the answer instead of generating a new one
Why C? And why not the others?
Correct answer: C. It first trains a separate reward model to predict which of two outputs a human would prefer, using the collected pairwise comparisons as training data, and then uses reinforcement learning (commonly PPO) to fine-tune the language model's policy to maximize the score that reward model assigns to its outputs
RLHF's standard pipeline trains a reward model on the collected pairwise human preference comparisons so it learns to score outputs the way a human annotator would, and then uses that learned reward model as the optimization signal for reinforcement learning -- typically PPO -- to update the language model's own policy so it produces outputs the reward model scores highly, aligning it with human preferences beyond the original pretraining objective. The option describing direct supervised editing of weights using preference labels as next-token targets is wrong because preference labels indicate which of two full outputs is better, not what the next token should be, so they cannot serve as ordinary supervised targets without the intermediate reward-modeling and RL steps. The option describing corpus filtering and retraining from scratch is wrong because RLHF fine-tunes the existing pretrained policy using reinforcement learning; it does not discard and rebuild the pretraining corpus. The option describing a retrieval index is wrong because RLHF changes the model's own generative behavior through training, rather than replacing generation with retrieval of past outputs.
Source: Ouyang et al., 'Training language models to follow instructions with human feedback' (InstructGPT), arXiv:2203.02155 (2022)