When evaluating an autoregressive language model on the same held-out test set, Model A gets a perplexity of 18.4 and Model B gets a perplexity of 42.1. Holding the tokenizer and test set fixed, what does this comparison indicate about the two models?
- Model B is better, because a higher perplexity score means the model assigns more probability mass to a wider range of plausible continuations, which reflects greater linguistic flexibility
- Neither score is meaningful on its own; perplexity only becomes interpretable once the model has also been evaluated on a downstream task such as translation or summarization
- Model A is better at predicting this test text than Model B, because perplexity is derived from the model's average per-token negative log-likelihood, and a lower value means the model assigned higher probability, on average, to the tokens that actually occurred
- Model A used a larger training corpus than Model B, since perplexity is primarily a measure of how much data a model was trained on rather than of its predictive accuracy
Why C? And why not the others?
Correct answer: C. Model A is better at predicting this test text than Model B, because perplexity is derived from the model's average per-token negative log-likelihood, and a lower value means the model assigned higher probability, on average, to the tokens that actually occurred
Perplexity is computed by exponentiating the model's average per-token negative log-likelihood on the test text, so it directly measures how surprised the model is by the tokens that actually appear; a lower value means the model consistently assigned higher probability to the real continuation, which is exactly what happens for Model A here. The option claiming a higher score reflects 'flexibility' has the direction backwards: perplexity punishes a model for spreading probability away from the tokens that actually occurred, so a higher value reflects worse prediction, not a virtue. The option claiming perplexity is meaningless without a downstream task is wrong because perplexity is itself a standard, self-contained intrinsic evaluation of next-token prediction quality, independent of any downstream application. The option attributing the difference to training-corpus size is wrong because perplexity is computed purely from the model's output probabilities on the given test set; it says nothing directly about how much data the model was trained on, only about its predictive accuracy on that specific text.
Source: Jelinek, Mercer, Bahl & Baker, 'Perplexity—a measure of the difficulty of speech recognition tasks,' Journal of the Acoustical Society of America (1977), the original definition of perplexity as an evaluation metric for language models