Zhao et al. (2021), "Calibrate Before Use: Improving Few-Shot Performance of Language Models," identify "recency bias" as one cause of instability in few-shot prompting. What does recency bias describe?
- The model's tendency to disproportionately predict whichever label appeared in the example placed nearest the end of the few-shot prompt, regardless of the true input
- The tendency of a model's accuracy to decline over time as new versions of the model are released
- The model favoring the most recently published research papers when asked to cite sources
- The tendency to give more weight to the very first example in a few-shot prompt while ignoring later examples
Correct answer: A. The model's tendency to disproportionately predict whichever label appeared in the example placed nearest the end of the few-shot prompt, regardless of the true input
Zhao et al. show that few-shot LLM predictions are systematically biased toward the answer of whichever demonstration example sits closest to the query at the end of the prompt -- what they term recency bias -- alongside majority label bias, which favors whichever label is most frequent among the examples, and common token bias; they propose contextual calibration to correct for these effects. The second and third options describe unrelated phenomena, model performance drift across product releases and citation preferences, that this paper does not study or address. The fourth option describes the opposite of recency bias: it describes a primacy effect toward the first example, whereas the paper's finding is specifically a bias toward the last, most recent example in the prompt, not the earliest one.
Source: Zhao, Wallace, Feng, Klein, Singh, "Calibrate Before Use: Improving Few-Shot Performance of Language Models" (ICML 2021), arXiv:2102.09690