A machine translation system produces a fluent, accurate translation that a human judge rates as excellent, but it uses different word choices and sentence structure than the single reference translation available in the test set. What is the most likely practical consequence for this output's BLEU score, and why?
- Its BLEU score will likely be low, because BLEU is computed from n-gram overlap between the candidate and the reference translation(s), so a valid paraphrase that shares few overlapping n-grams with the single reference is penalized even though it is fluent and accurate
- Its BLEU score will be high, because BLEU uses a semantic similarity model to detect that the paraphrase conveys the same meaning as the reference, regardless of surface wording
- Its BLEU score will be unaffected by the reference wording, because BLEU is computed only from the candidate sentence's internal grammaticality and fluency, without reference to any reference translation
- Its BLEU score will be low, because BLEU applies a fixed penalty whenever the candidate sentence is longer than the reference sentence, independent of any n-gram overlap
Why A? And why not the others?
Correct answer: A. Its BLEU score will likely be low, because BLEU is computed from n-gram overlap between the candidate and the reference translation(s), so a valid paraphrase that shares few overlapping n-grams with the single reference is penalized even though it is fluent and accurate
BLEU scores a candidate translation by counting how many of its n-grams (contiguous word sequences) also appear in the reference translation(s), modified by a brevity penalty for candidates shorter than the reference; it has no mechanism for recognizing that a differently-worded sentence expresses the same meaning, so a fluent, accurate paraphrase that reuses few of the reference's exact word sequences will score low despite being a good translation. The option claiming BLEU uses a semantic similarity model is wrong because BLEU is a purely lexical, n-gram-overlap metric with no learned semantic component. The option claiming BLEU ignores the reference entirely and scores only the candidate's own grammaticality is wrong because reference comparison is the entire basis of the metric; a candidate cannot be scored without at least one reference to compare against. The option attributing the low score to sentence length is wrong because BLEU's brevity penalty specifically targets candidates that are shorter than the reference, not longer, and the scenario describes a wording difference rather than a length difference.
Source: Papineni, Roukos, Ward & Zhu, 'BLEU: a Method for Automatic Evaluation of Machine Translation,' Proceedings of ACL 2002