A retrieval system is evaluated on a query for which there are 10 truly relevant passages somewhere in the corpus. The system returns 20 passages for that query, and 8 of those 20 are among the 10 truly relevant passages. What are this query's recall@20 and precision@20?
- Recall@20 = 8/10 = 0.8, because 8 of the 10 relevant passages were retrieved; precision@20 = 8/20 = 0.4, because only 8 of the 20 returned passages were relevant
- Recall@20 = 8/20 = 0.4, because 8 of the 20 returned passages were relevant; precision@20 = 8/10 = 0.8, because 8 of the 10 relevant passages were retrieved
- Recall@20 and precision@20 are both 8/18 = 0.44, since the two missed relevant passages and the twelve irrelevant returned passages should be pooled into a single combined denominator
- Recall@20 cannot be computed from the numbers given because it requires knowing the total size of the corpus, whereas precision@20 can be computed and equals 8/20 = 0.4
Why A? And why not the others?
Correct answer: A. Recall@20 = 8/10 = 0.8, because 8 of the 10 relevant passages were retrieved; precision@20 = 8/20 = 0.4, because only 8 of the 20 returned passages were relevant
Recall@k is defined as the number of truly relevant items found within the top k results divided by the total number of truly relevant items that exist for that query, which here is 8 divided by 10, or 0.8; precision@k is defined as the number of truly relevant items found within the top k results divided by k itself, which here is 8 divided by 20, or 0.4. The option that swaps the two formulas -- dividing the relevant-and-retrieved count by 20 to get recall and by 10 to get precision -- is the classic error of confusing these two definitions and applying each one's arithmetic to the wrong metric. The option pooling everything into a denominator of 18 is wrong because neither metric's standard definition combines missed relevant passages and irrelevant retrieved passages into a single shared denominator; each metric has its own distinct denominator. The option claiming recall cannot be computed is wrong because recall's denominator is the total number of truly relevant passages for the query, a quantity already given as 10 in the scenario, not the unrelated and much larger size of the entire corpus.
Source: Manning, Raghavan & Schutze, 'Introduction to Information Retrieval' (2008), Chapter 8, Evaluation in Information Retrieval, https://nlp.stanford.edu/IR-book/