Evaluation, Testing & Safety

Why C? And why not the others?

Correct answer: C. Model A is better at predicting this test text than Model B, because perplexity is derived from the model's average per-token negative log-likelihood, and a lower value means the model assigned higher probability, on average, to the tokens that actually occurred

Perplexity is computed by exponentiating the model's average per-token negative log-likelihood on the test text, so it directly measures how surprised the model is by the tokens that actually appear; a lower value means the model consistently assigned higher probability to the real continuation, which is exactly what happens for Model A here. The option claiming a higher score reflects 'flexibility' has the direction backwards: perplexity punishes a model for spreading probability away from the tokens that actually occurred, so a higher value reflects worse prediction, not a virtue. The option claiming perplexity is meaningless without a downstream task is wrong because perplexity is itself a standard, self-contained intrinsic evaluation of next-token prediction quality, independent of any downstream application. The option attributing the difference to training-corpus size is wrong because perplexity is computed purely from the model's output probabilities on the given test set; it says nothing directly about how much data the model was trained on, only about its predictive accuracy on that specific text.

Source: Jelinek, Mercer, Bahl & Baker, 'Perplexity—a measure of the difficulty of speech recognition tasks,' Journal of the Acoustical Society of America (1977), the original definition of perplexity as an evaluation metric for language models

Why A? And why not the others?

Correct answer: A. Its BLEU score will likely be low, because BLEU is computed from n-gram overlap between the candidate and the reference translation(s), so a valid paraphrase that shares few overlapping n-grams with the single reference is penalized even though it is fluent and accurate

BLEU scores a candidate translation by counting how many of its n-grams (contiguous word sequences) also appear in the reference translation(s), modified by a brevity penalty for candidates shorter than the reference; it has no mechanism for recognizing that a differently-worded sentence expresses the same meaning, so a fluent, accurate paraphrase that reuses few of the reference's exact word sequences will score low despite being a good translation. The option claiming BLEU uses a semantic similarity model is wrong because BLEU is a purely lexical, n-gram-overlap metric with no learned semantic component. The option claiming BLEU ignores the reference entirely and scores only the candidate's own grammaticality is wrong because reference comparison is the entire basis of the metric; a candidate cannot be scored without at least one reference to compare against. The option attributing the low score to sentence length is wrong because BLEU's brevity penalty specifically targets candidates that are shorter than the reference, not longer, and the scenario describes a wording difference rather than a length difference.

Source: Papineni, Roukos, Ward & Zhu, 'BLEU: a Method for Automatic Evaluation of Machine Translation,' Proceedings of ACL 2002

Why D? And why not the others?

Correct answer: D. This is position bias in LLM-as-judge evaluation; a standard mitigation is to run each comparison twice with the two responses' positions swapped, and only count a response as preferred if it wins in both orderings (or average the results across both orderings)

This is position bias, a well-documented weakness of using an LLM as an automated judge: the judge's verdict shifts based on which slot a response is placed in, independent of the response's actual content, which is exactly what the swap-and-re-judge test in the scenario reveals. The standard mitigation is to run each pairwise comparison in both orderings and only treat a response as the winner if it wins regardless of which slot it occupies (or to average the two orderings' results), which cancels out the ordering effect. The option calling this a sign the judged model is broken is wrong because the inconsistency described is in the judge's verdicts, not in the model being judged, which produced the same two responses both times. The option calling this benchmark contamination is wrong because contamination refers to test data leaking into training data, not to a judge's sensitivity to answer placement. The option claiming no mitigation is needed is wrong because treating an artifact of response placement as a genuine quality signal would systematically bias the evaluation toward whichever response happens to be listed first.

Source: Zheng et al., 'Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,' arXiv:2306.05685 (2023) -- documents position bias, verbosity bias, and self-enhancement bias in LLM-as-judge evaluation

Why B? And why not the others?

Correct answer: B. This threat is called benchmark (data) contamination; one concrete detection technique is searching the pretraining corpus for high n-gram overlap with the benchmark's test examples, or checking for embedded 'canary string' markers placed in the benchmark specifically to detect this

Benchmark (data) contamination occurs when test examples, or near-duplicates of them, are present in a model's pretraining corpus, which inflates its benchmark score by letting it partially memorize answers rather than generalize; a standard way to check for it is to measure n-gram overlap between the pretraining corpus and the benchmark's questions, or to check whether the model reproduces 'canary strings' -- unique marker text some benchmark creators deliberately insert so contamination can be detected later. The option describing catastrophic forgetting is wrong because that term refers to a model losing previously learned capability after further training, not to test data leaking into training data. The option describing mode collapse is wrong because that refers to a generative model producing overly repetitive or non-diverse outputs, which is unrelated to whether test data leaked into training. The option describing reward hacking is wrong because that refers to a policy exploiting flaws in a reward model during reinforcement learning, not to overlap between benchmark and pretraining text.

Source: Brown et al., 'Language Models are Few-Shot Learners' (GPT-3 paper), arXiv:2005.14165 (2020), Appendix C, which measures benchmark contamination via n-gram overlap between the pretraining corpus and benchmark test sets

Why A? And why not the others?

Correct answer: A. This is red-teaming; unlike routine functional testing, which checks whether the model performs its intended tasks correctly, red-teaming specifically and adversarially searches for inputs that cause unsafe or policy-violating behavior, deliberately trying to break the model's safety guardrails rather than verify normal functionality

Red-teaming is the practice of deliberately and adversarially probing a model with creative attacks before deployment specifically to surface harmful or policy-violating behavior, which is fundamentally different in intent from routine functional testing that simply checks whether the model does its intended job correctly; the goal in red-teaming is to break the model's safety guardrails on purpose so the weaknesses can be fixed first. The option describing A/B testing is wrong because that method compares average outcomes between two model versions shown to ordinary users under normal conditions, with no adversarial intent to provoke unsafe behavior. The option describing unit testing is wrong because it reduces the distinction to who writes the test cases rather than to the adversarial, safety-focused intent that actually defines red-teaming. The option describing regression testing is wrong because it reduces the distinction to timing relative to deployment rather than to the adversarial nature of the testing itself, and regression testing is about confirming old functionality still works, not about discovering new harmful behaviors.

Source: Ganguli et al., 'Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned,' arXiv:2202.03286 (Anthropic, 2022)

Why B? And why not the others?

Correct answer: B. The first is a direct prompt injection (or jailbreak), where the application's own user is the adversary crafting the input; the second is an indirect prompt injection, where the user is trusted but the agent processes third-party content -- such as a webpage, email, or tool result -- that itself contains adversarial instructions

The distinction turns on who the adversary is: in the first case the application's own user is deliberately crafting the malicious input themselves, which is a direct prompt injection or jailbreak, while in the second case the user is trusted and simply asked for a legitimate summary, but the agent ingests third-party content -- the webpage -- that itself carries adversarial instructions, which is an indirect prompt injection. The option treating both as the same 'jailbreaking' attack is wrong because collapsing the two erases the difference in who controls the adversarial input, which changes what mitigations are appropriate (screening user input versus treating tool/webpage content as untrusted). The option that swaps the two labels is wrong because it inverts the actual basis for the distinction, which is the source of the adversarial content, not how 'directly' the text reaches the model. The option invoking data poisoning is wrong because data poisoning specifically means corrupting a model's training data, not injecting adversarial text into its input at inference time, which is what both scenarios describe.

Source: Anthropic, 'Mitigate jailbreaks and prompt injections,' platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaks -- distinguishes direct prompt injection/jailbreaks (user is the adversary) from indirect prompt injection (trusted user, adversarial third-party content)

Why C? And why not the others?

Correct answer: C. It first trains a separate reward model to predict which of two outputs a human would prefer, using the collected pairwise comparisons as training data, and then uses reinforcement learning (commonly PPO) to fine-tune the language model's policy to maximize the score that reward model assigns to its outputs

RLHF's standard pipeline trains a reward model on the collected pairwise human preference comparisons so it learns to score outputs the way a human annotator would, and then uses that learned reward model as the optimization signal for reinforcement learning -- typically PPO -- to update the language model's own policy so it produces outputs the reward model scores highly, aligning it with human preferences beyond the original pretraining objective. The option describing direct supervised editing of weights using preference labels as next-token targets is wrong because preference labels indicate which of two full outputs is better, not what the next token should be, so they cannot serve as ordinary supervised targets without the intermediate reward-modeling and RL steps. The option describing corpus filtering and retraining from scratch is wrong because RLHF fine-tunes the existing pretrained policy using reinforcement learning; it does not discard and rebuild the pretraining corpus. The option describing a retrieval index is wrong because RLHF changes the model's own generative behavior through training, rather than replacing generation with retrieval of past outputs.

Source: Ouyang et al., 'Training language models to follow instructions with human feedback' (InstructGPT), arXiv:2203.02155 (2022)

Why D? And why not the others?

Correct answer: D. The claims-supported-by-retrieved-passages score is faithfulness; it specifically measures whether the claims made in the generated answer can be traced back to and supported by the retrieved context, independent of whether that context was relevant to the question or whether the answer addresses the question asked

Faithfulness (also called groundedness) specifically measures whether each claim in the generated answer can be traced back to, and is supported by, the passages the retrieval step actually returned -- it is a check on whether the generator invented or distorted claims beyond what its retrieved evidence supports, and it is logically independent of the other two scores: an answer can be grounded in its context yet fail to address the question, or address the question well while retrieval itself was off-topic. The option labeling the retrieval-relevance score as faithfulness is wrong because relevance of the retrieved passages to the question says nothing about whether the generated answer's specific claims are actually supported by those passages. The option labeling the answer-relevance score as faithfulness is wrong for the same reason in reverse: directly addressing the question's wording says nothing about whether the answer's claims are grounded in retrieved evidence rather than fabricated. The option claiming all three scores are mathematically the same quantity is wrong because they are defined to capture distinct failure modes -- poor retrieval, off-topic answers, and ungrounded claims can each occur independently of the others.

Source: Es et al., 'RAGAS: Automated Evaluation of Retrieval Augmented Generation,' arXiv:2309.15217 (2023) -- defines faithfulness, answer relevance, and context relevance/precision as distinct RAG evaluation metrics

Why A? And why not the others?

Correct answer: A. Relative judgments -- deciding which of two responses is better -- are generally easier for humans to make consistently than assigning an absolute numeric score, so pairwise comparisons tend to produce less noisy, better-calibrated signal for ranking models against each other than independently-collected absolute ratings, which vary more across raters and over time

Pairwise comparisons are considered more reliable because judging which of two responses is better is a simpler, more consistent task for humans than assigning an absolute numeric score, since absolute scales suffer from raters anchoring differently, drifting over time, and disagreeing about what a given number means, whereas a relative 'which is better' judgment sidesteps most of that calibration problem; aggregating many such pairwise votes into an Elo-style rating is exactly how platforms like Chatbot Arena rank models. The option claiming absolute ratings are 'mathematically impossible' to aggregate is wrong because averaging or otherwise combining independent ratings is entirely possible, just noisier and harder to calibrate across raters than pairwise comparison; it is also wrong that pairwise votes need no aggregation, since converting many pairwise outcomes into an Elo rating is itself an aggregation step. The option attributing the advantage to collection speed and vote count alone is wrong because it ignores that the reliability gain comes from the nature of the judgment task itself, not merely from how fast or how many votes are collected. The option restricting absolute scales to machine translation is wrong because absolute rating scales are used across many evaluation tasks; they are not confined to translation.

Source: Chiang et al., 'Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference,' arXiv:2403.04132 (2024)

Why B? And why not the others?

Correct answer: B. It can be called on both standalone user-submitted input and on text generated by a model, returning a flagged status plus per-category violation flags and confidence scores; per OpenAI's own documentation, the underlying model still generates its output normally, and the moderation results are meant to be reviewed before that output is shown to a user or acted on

OpenAI's Moderation API can classify standalone text a user submits as well as text a model has already generated, returning a binary flagged status, per-category violation flags, and confidence scores for each of its content categories; the documentation is explicit that the underlying generative model still produces its output normally and that the moderation results should be reviewed before that output is shown to a user or used to trigger a downstream action, meaning it is a review layer rather than a generation-time filter. The option claiming it replaces the need for the underlying model's own safety training is wrong because it is described as an additional review step applied around generation, not a substitute for the model's own alignment. The option restricting it to pre-generation user input only is wrong because the documentation explicitly supports scoring model-generated text as well. The option describing it as a paid, metered endpoint is wrong because OpenAI's documentation states the moderation endpoint is free to use.

Source: OpenAI, 'Moderation' guide, platform.openai.com/docs/guides/moderation (redirects to developers.openai.com/api/docs/guides/moderation) -- describes the omni-moderation-latest model, its categories, its use on both input and generated output, and that it is free to use