A team building an agent finds two different attack patterns during security testing: in one, a user of the application directly crafts an unusual prompt trying to get the agent to ignore its instructions; in the other, a trusted user asks the agent to summarize a webpage, and the webpage's own text contains hidden instructions trying to redirect the agent. Using the threat-model distinction drawn in Anthropic's guidance for mitigating these attacks, how are the two cases distinguished?
- Both are the same attack, jailbreaking, because in both cases the ultimate effect is that the agent's system prompt is overridden, regardless of who supplied the adversarial text
- The first is a direct prompt injection (or jailbreak), where the application's own user is the adversary crafting the input; the second is an indirect prompt injection, where the user is trusted but the agent processes third-party content -- such as a webpage, email, or tool result -- that itself contains adversarial instructions
- The first is indirect prompt injection, because the user's prompt is only indirectly related to the agent's task, while the second is direct prompt injection, because the webpage content directly reaches the model
- The first is a data poisoning attack, because it targets the model's training data, while the second is a prompt injection attack, because it targets the model's inference-time input
Why B? And why not the others?
Correct answer: B. The first is a direct prompt injection (or jailbreak), where the application's own user is the adversary crafting the input; the second is an indirect prompt injection, where the user is trusted but the agent processes third-party content -- such as a webpage, email, or tool result -- that itself contains adversarial instructions
The distinction turns on who the adversary is: in the first case the application's own user is deliberately crafting the malicious input themselves, which is a direct prompt injection or jailbreak, while in the second case the user is trusted and simply asked for a legitimate summary, but the agent ingests third-party content -- the webpage -- that itself carries adversarial instructions, which is an indirect prompt injection. The option treating both as the same 'jailbreaking' attack is wrong because collapsing the two erases the difference in who controls the adversarial input, which changes what mitigations are appropriate (screening user input versus treating tool/webpage content as untrusted). The option that swaps the two labels is wrong because it inverts the actual basis for the distinction, which is the source of the adversarial content, not how 'directly' the text reaches the model. The option invoking data poisoning is wrong because data poisoning specifically means corrupting a model's training data, not injecting adversarial text into its input at inference time, which is what both scenarios describe.
Source: Anthropic, 'Mitigate jailbreaks and prompt injections,' platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaks -- distinguishes direct prompt injection/jailbreaks (user is the adversary) from indirect prompt injection (trusted user, adversarial third-party content)