In short
Red teaming is the targeted, adversarial testing of a system: a team slips into the role of the attacker and tries, with every realistic means, to deceive the system, misuse it or make it misbehave before real attackers or everyday use do. For AI systems, this means prompts, inputs and scenarios designed to push the model outside its guardrails, systematically and in a documented way.
The difference from normal testing lies in the attitude. A classic test checks whether the system does what it is supposed to do. Red teaming checks what the system can be made to do when someone seriously sets out to achieve it. For AI agents that independently write emails, change records or trigger orders, this check is not a security exercise for corporations but the prerequisite for a responsible go-live.
Origins: from the military via IT security to AI
The term comes from military war games of the Cold War: a "red team" simulated the enemy in order to test its own plans against real opposition, while a "blue team" defended. IT security adopted the model: red teams attack networks, applications and employees (for example via phishing), blue teams detect and defend. This tradition is also the source of the distinction from the penetration test: a pentest checks defined systems within a defined scope for known vulnerabilities, while red teaming simulates a real adversary with a free choice of targets.
With generative AI, the term gained a third career. Model providers test their systems adversarially before release, and companies introducing AI applications adopt the method for their own context of use: what is tested is not the model as such but the specific application with its prompts, data sources, tools and permissions.
What makes red teaming special for AI systems
Classic software behaves deterministically: same input, same output. A language model does not. It can react differently to the same input, and it can be attacked through natural language, not just through code. The attack surface is therefore every point at which text enters the system: user input, emails, tickets, documents, web pages, even tool descriptions.
The most important attack classes that an AI red team plays through:
- Direct prompt injection: the attacker writes their instructions directly into the input ("Ignore your rules and …"). OWASP lists prompt injection as LLM01:2025, in first place among the risks for LLM applications.
- Indirect prompt injection: the instruction is hidden in content that the system processes, such as an email read by a support agent or a web page visited by a research agent. For agents this is the most dangerous vector, because the injection can trigger actions.
- Jailbreaks: circumventing the behavioural rules through role play, nesting or gradual softening, so that the system does or says things that its guardrails forbid.
- Tool misuse and excessive agency: the attacker gets an agent to use its tools against the operator: exporting data, sending mass emails, deleting records. The more permissions the agent has, the greater the possible damage.
- Data exfiltration: attempts to coax confidential content out of system prompts, connected knowledge sources or earlier conversations.
No single measure helps against these attacks; what helps is defence in layers: restrictive permissions so that a compromised agent can do little harm, technical guardrails for recognisable patterns, human approvals (human-in-the-loop) for critical actions, and red teaming, which checks whether these layers hold up together. It is precisely this interplay that is under test: a single filter can almost always be circumvented, a well-built chain rarely.
How red teaming before go-live works
In practice, a four-step process has become established that mid-sized companies without their own security team can also implement:
| Step | Content | Artefact |
|---|---|---|
| 1. Define scenarios | Regular cases, edge cases and attacks for the specific context of use, including the attack classes above | Scenario catalogue |
| 2. Set pass/fail criteria | Define in advance what "passed" means, e.g. zero write actions without approval, clean escalation for every attack | Criteria list |
| 3. Attack and log | Play through scenarios, document behaviour, classify findings by severity and fix them | Findings log |
| 4. Convert into an eval suite | The scenarios become a repeatable test suite that runs again with every prompt change and every model switch | Eval suite / regression tests |
Step 4 is the one most often skipped and the most important. Red teaming as a one-off exercise becomes outdated with the first change: an update of the underlying model can shift behaviour without anyone having touched their own code. Only as a permanent eval suite does the test become a control instrument, as robust AI governance requires.
Where do the scenarios come from?
The scenario catalogue draws on three sources. First, public threat catalogues: the OWASP Top 10 for LLM applications and the agentic AI catalogue provide the attack classes that every system should cover. Second, your own processes: what would be the most expensive mistake in this specific deployment? For a procurement agent, the scenarios differ from those for a support agent. Third, documented incidents elsewhere: public cases such as the Replit incident of July 2025, in which a coding agent deleted a production database despite a code freeze, can almost always be translated into a question for your own system: could this happen to us, and would anyone see it in time?
Typical patterns of findings
Across projects, certain classes of findings recur in AI red teaming. At the top: agents with far more permissions than their task requires, usually because an existing API key with full access was reused. Behind that: indirect injections via content that nobody had considered an input channel, such as email signatures or free-text fields in forms. And finally, missing escalation rules: the system recognises a contradiction or a suspicious instruction but has no defined way to involve a human, and therefore decides for itself. All three patterns are organisational defects, not model weaknesses, and that is precisely why you only find them if you test the overall system rather than the model alone.
Who should test?
Preferably not only the people who built the system. Anyone who sees the agent every day demonstrably tests it ever more gently. Anthropic documented this in its own long-term experiment Project Vend: an agent ran a small office kiosk there, and internal red teaming dozed off over time because the agent had become part of everyday life. Anthropic then handed the kiosk over to reporters from the Wall Street Journal, explicitly as a hostile environment outside its own control. The lesson for every company: for the assessment, bring in people who want to break the system, such as colleagues from another department, an external service provider or a specialised AI agency.
Red teaming, pentests, evals: the distinction
Three terms that are often mixed up in AI projects: a penetration test checks the classic attack surface (network, server, application) for technical vulnerabilities, typically with a defined scope. Evals are repeatable quality tests for AI behaviour: does the system carry out its tasks correctly, how often does it escalate, what is the error rate? Red teaming is the adversarial view: it looks specifically for ways to misuse the system and then feeds its findings into the evals, so that every attack route found is retained permanently as a test case and a gap that has been closed does not reopen unnoticed. Productive agents need all three, in this order: a pentest for the infrastructure, red teaming before go-live, evals in continuous operation.
On the question of roles: internal or external is not an either-or decision. An internal team knows the processes and knows which mistake really hurts; external testers bring fresh attack ideas and are not blinkered by routine. A proven division of labour is for the internal team to build the scenario catalogue from its own processes and run the eval suite, while an external view is added for the go-live test and the annual refresh. In both cases, what matters is a clear mandate with defined limits: which systems may be attacked, with which data, in which time window, and who is informed if a test accidentally has an effect on production systems.
For agentic systems, OWASP has since expanded the methodology: the catalogue "Agentic AI – Threats and Mitigations" from the OWASP initiative for generative AI describes threats that only arise through tools, memory and autonomy, such as tool poisoning or misuse of agent permissions, and is a good basis for your own scenario catalogue, especially for a multi-agent system with several roles involved.
Frequently asked questions about red teaming
Do small companies really need red teaming?
As soon as an AI system has write access to business systems or talks directly to customers: yes. The scope scales with the risk. For an internal assistant, a few hours of structured adversarial testing with a scenario catalogue are enough; for an agent with payment or deletion rights, serious red teaming with external testers belongs before every go-live.
How often should red teaming be repeated?
The full red teaming before go-live, and the resulting eval suite with every relevant change: a new prompt, new tools, new data sources and above all every update of the underlying model. In addition, a periodic refresh of the scenario catalogue is worthwhile, because new attack techniques emerge.
What does red teaming for an AI system cost a mid-sized company?
The honest answer: significantly less than the incident it prevents. The effort depends on the permissions and reach of the system, from a few person-days for an internal assistant to a programme of several weeks for agents with far-reaching permissions. The biggest cost lever is reuse: a scenario catalogue built once supports every further AI project.
Can red teaming be automated?
Partly. There are tools that run known attack patterns against a system automatically, and LLMs can also be used as attacker simulators. However, automation does not replace the human creativity with which real attackers exploit context. A proven combination: automated suites for breadth and regression, humans for new, context-specific attacks.
Is red teaming required by law?
Not explicitly for most business applications. However, for high-risk systems (from December 2027) the EU AI Act requires risk management and accuracy and robustness requirements that can hardly be demonstrated reliably without adversarial testing. Red teaming is therefore the practical answer from a regulatory point of view too, not just the technically sensible one.