Before releasing a new model, a safety team assembles a group of people whose specific job is to deliberately try to provoke the model into producing harmful, disallowed, or policy-violating outputs, using creative and adversarial prompts, so the team can find and fix weaknesses before real users encounter them. What is this practice called, and how does it differ from routine functional testing?
- This is red-teaming; unlike routine functional testing, which checks whether the model performs its intended tasks correctly, red-teaming specifically and adversarially searches for inputs that cause unsafe or policy-violating behavior, deliberately trying to break the model's safety guardrails rather than verify normal functionality
- This is A/B testing; like routine functional testing, it measures average task success rate across two model versions shown to random subsets of real users, without any adversarial intent
- This is unit testing; it differs from routine functional testing only in that the test cases are written by an external team rather than the model's own developers
- This is regression testing; it differs from routine functional testing only in that it is run after every deployment rather than before the first one
Why A? And why not the others?
Correct answer: A. This is red-teaming; unlike routine functional testing, which checks whether the model performs its intended tasks correctly, red-teaming specifically and adversarially searches for inputs that cause unsafe or policy-violating behavior, deliberately trying to break the model's safety guardrails rather than verify normal functionality
Red-teaming is the practice of deliberately and adversarially probing a model with creative attacks before deployment specifically to surface harmful or policy-violating behavior, which is fundamentally different in intent from routine functional testing that simply checks whether the model does its intended job correctly; the goal in red-teaming is to break the model's safety guardrails on purpose so the weaknesses can be fixed first. The option describing A/B testing is wrong because that method compares average outcomes between two model versions shown to ordinary users under normal conditions, with no adversarial intent to provoke unsafe behavior. The option describing unit testing is wrong because it reduces the distinction to who writes the test cases rather than to the adversarial, safety-focused intent that actually defines red-teaming. The option describing regression testing is wrong because it reduces the distinction to timing relative to deployment rather than to the adversarial nature of the testing itself, and regression testing is about confirming old functionality still works, not about discovering new harmful behaviors.
Source: Ganguli et al., 'Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned,' arXiv:2202.03286 (Anthropic, 2022)