A public platform for comparing chatbot quality shows users two anonymous model responses to the same prompt side by side and asks only 'which response is better,' rather than asking users to rate each response independently on, say, a 1-5 scale. The platform then aggregates these pairwise votes into a single ranking (an Elo-style rating) for each model. Why is this pairwise-comparison approach generally considered more reliable than collecting independent absolute ratings for ranking relative model quality?
- Relative judgments -- deciding which of two responses is better -- are generally easier for humans to make consistently than assigning an absolute numeric score, so pairwise comparisons tend to produce less noisy, better-calibrated signal for ranking models against each other than independently-collected absolute ratings, which vary more across raters and over time
- Absolute ratings are mathematically impossible to aggregate across different raters, while pairwise comparisons require no aggregation step at all before a ranking can be produced
- Pairwise comparisons take less time for the platform to collect than absolute ratings, and evaluation validity is determined entirely by how many total votes are collected, regardless of how the question is framed
- Absolute rating scales are only usable for evaluating machine translation quality specifically, so any other evaluation task must default to pairwise comparison instead
Why A? And why not the others?
Correct answer: A. Relative judgments -- deciding which of two responses is better -- are generally easier for humans to make consistently than assigning an absolute numeric score, so pairwise comparisons tend to produce less noisy, better-calibrated signal for ranking models against each other than independently-collected absolute ratings, which vary more across raters and over time
Pairwise comparisons are considered more reliable because judging which of two responses is better is a simpler, more consistent task for humans than assigning an absolute numeric score, since absolute scales suffer from raters anchoring differently, drifting over time, and disagreeing about what a given number means, whereas a relative 'which is better' judgment sidesteps most of that calibration problem; aggregating many such pairwise votes into an Elo-style rating is exactly how platforms like Chatbot Arena rank models. The option claiming absolute ratings are 'mathematically impossible' to aggregate is wrong because averaging or otherwise combining independent ratings is entirely possible, just noisier and harder to calibrate across raters than pairwise comparison; it is also wrong that pairwise votes need no aggregation, since converting many pairwise outcomes into an Elo rating is itself an aggregation step. The option attributing the advantage to collection speed and vote count alone is wrong because it ignores that the reliability gain comes from the nature of the judgment task itself, not merely from how fast or how many votes are collected. The option restricting absolute scales to machine translation is wrong because absolute rating scales are used across many evaluation tasks; they are not confined to translation.
Source: Chiang et al., 'Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference,' arXiv:2403.04132 (2024)