Gathering large volumes of high-quality human preference labels for RLHF is expensive, so Lee et al. studied replacing the human-labeled preference data used to train a reward model with preference labels generated by an off-the-shelf large language model instead (RLAIF), then compared the resulting policies across summarization and dialogue tasks using human evaluators. What did the study actually find?
- RLAIF only matched human-evaluated RLHF performance when the LLM generating the preference labels was substantially larger and more capable than the policy model being trained, and performed far worse whenever the labeler was the same size as the policy
- RLAIF consistently and substantially underperformed both RLHF and a plain supervised-fine-tuned baseline across every task tested, showing that AI-generated preference labels cannot substitute for human-labeled ones
- RLAIF achieved performance comparable to RLHF as judged by human evaluators across the tasks tested, including cases where the labeling model was the same size as the policy being trained or even the exact same checkpoint, and RLAIF outperformed the supervised-fine-tuned baseline in these comparisons
- RLAIF eliminated the need for a reward model and the reinforcement-learning optimization step entirely in every configuration tested, since the AI-generated labels were used to edit the policy's weights directly without any RL algorithm
Why C? And why not the others?
Correct answer: C. RLAIF achieved performance comparable to RLHF as judged by human evaluators across the tasks tested, including cases where the labeling model was the same size as the policy being trained or even the exact same checkpoint, and RLAIF outperformed the supervised-fine-tuned baseline in these comparisons
Lee et al. found that a reward model trained on preference labels generated by an off-the-shelf LLM, rather than by human annotators, produced policies that human evaluators rated comparably to policies trained with standard human-labeled RLHF across the summarization and dialogue tasks tested, and this held even in the more demanding case where the model generating the labels was the same size as the policy being trained, or the identical checkpoint as the policy's own starting point; RLAIF also beat a supervised-fine-tuned baseline in these comparisons. The option requiring the labeler to be strictly larger and more capable is wrong because the study specifically highlights comparable performance even at equal size or an identical checkpoint. The option claiming RLAIF substantially underperformed both RLHF and the supervised baseline is the opposite of the paper's finding. The option claiming the reward model and the RL step were eliminated entirely in every tested configuration describes a separate variant the paper calls direct-RLAIF, not the core RLAIF setup evaluated in most of the study's main comparisons, which still trains a reward model and runs RL against it, just with AI-generated rather than human-generated preference labels.
Source: Lee et al., 'RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback' (arXiv:2309.00267, 2023)