A research team reports that their new model scores surprisingly high on a widely-used public benchmark, higher than models with far more parameters. Before trusting this result, an independent reviewer wants to rule out one specific and common threat to benchmark validity: that some of the benchmark's test questions, or close paraphrases of them, appeared in the model's pretraining data. What is this threat called, and what is one concrete technique used to check for it?
- This threat is called catastrophic forgetting; it is checked by fine-tuning the model on an older task and confirming that performance on that older task does not drop
- This threat is called benchmark (data) contamination; one concrete detection technique is searching the pretraining corpus for high n-gram overlap with the benchmark's test examples, or checking for embedded 'canary string' markers placed in the benchmark specifically to detect this
- This threat is called mode collapse; it is checked by sampling many outputs from the model at high temperature and confirming that they remain diverse
- This threat is called reward hacking; it is checked by inspecting the model's reward model training data for label errors
Why B? And why not the others?
Correct answer: B. This threat is called benchmark (data) contamination; one concrete detection technique is searching the pretraining corpus for high n-gram overlap with the benchmark's test examples, or checking for embedded 'canary string' markers placed in the benchmark specifically to detect this
Benchmark (data) contamination occurs when test examples, or near-duplicates of them, are present in a model's pretraining corpus, which inflates its benchmark score by letting it partially memorize answers rather than generalize; a standard way to check for it is to measure n-gram overlap between the pretraining corpus and the benchmark's questions, or to check whether the model reproduces 'canary strings' -- unique marker text some benchmark creators deliberately insert so contamination can be detected later. The option describing catastrophic forgetting is wrong because that term refers to a model losing previously learned capability after further training, not to test data leaking into training data. The option describing mode collapse is wrong because that refers to a generative model producing overly repetitive or non-diverse outputs, which is unrelated to whether test data leaked into training. The option describing reward hacking is wrong because that refers to a policy exploiting flaws in a reward model during reinforcement learning, not to overlap between benchmark and pretraining text.
Source: Brown et al., 'Language Models are Few-Shot Learners' (GPT-3 paper), arXiv:2005.14165 (2020), Appendix C, which measures benchmark contamination via n-gram overlap between the pretraining corpus and benchmark test sets