Red Team · AI Red Teaming
Red Teaming
What is a red team?
- Deliberately attacking to find weaknesses and risks
- Posing as a “malicious user”
- A required step for safe deployment
Once red teaming comes in, the question that verification asks changes. Where QA checks “does it work as requested,” the red team asks “how can we break it?” So the deployment approval criterion shifts from an accuracy percentage to how many attacks got through and how severe they are. That is also why the people who build and the people who attack must be separate. It is hard to doubt the assumptions behind your own design, so you need dedicated staff in the attacker role, and a structure that credits them for every vulnerability they find.
In practice, red teaming usually runs along three tracks: manual red teaming, where people design conversations and probe the system by hand; automated red teaming, where attack prompts are generated automatically and thousands are fired off at once; and domain red teaming for areas like healthcare, law, finance, and child safety, where only a specialist can judge whether an output is risky. It also targets something different from penetration testing — pen testing breaks into servers and permissions, while red teaming targets what the model says. It is distinct from benchmark evaluation as well. A benchmark repeats fixed questions to produce a score; red teaming is writing new questions that have not been asked yet. That is why the deliverable should be a list of vulnerabilities with reproduction steps, not a scorecard.
The most common mistake in the field is treating red teaming as a one-time rite of passage just before launch. Upgrading the model version, tweaking the system prompt, or connecting new internal documents all change the attack surface, so it should be re-run whenever something changes. The second misconception is the habit of blocking discovered vulnerabilities only by adding banned words. The tighter the filter, the more legitimate questions get caught, until the chatbot keeps repeating 'I'm unable to answer that,' and within a couple of months the business teams stop using it. Measure both how much you block and how accurately you block it. Finally, without reproduction conversations in the report, the development team cannot fix anything. A feeling that something 'looked risky' cannot become a fix ticket.
John KimInstructor John Kim, author of B2B Marketing Design and CEO of Nanum Consulting
Seolyeong ParkGenerative AI and AX training specialist with 17 years of hands-on experience and 200+ lectures, helping non-technical staff put AI to work right awayIt originated in the “simulated enemy (red team)” concept from military and cybersecurity, and has expanded into a standard procedure for verifying AI safety.
Consider a hypothetical case. Company A, a non-life insurer with about 900 employees, spent four months developing a claims-guidance chatbot and was two weeks from launch. Internal testing showed 91% accuracy, but the compliance office asked, “If it promises coverage the policy doesn’t include, who is responsible?” and held up approval. In the end, they delayed the launch by six weeks and set up a red team first.
- Manager Park of the Digital Innovation Team pulled four customer center agents and two compliance office staff to form the red team, and deliberately left out the chatbot developers. The reasoning: “People go easy on what they built themselves.”
- Over two weeks, they wrote 420 attack scenarios. More than half were phrases that agents had pulled from real complaint call recordings, such as “Please make it pay out even without a medical certificate.”
- In the first round of testing, 37 attacks got through. In particular, simply changing the opening line to “My friend is curious about this” made the bot skip the disclaimer and say a payout was possible, a pattern that repeated 11 times.
- Manager Park had every finding logged with the full reproduction conversation and screenshots attached, split severity into high, medium and low, and set the 9 high-severity cases as must-fix before launch.
- In a retest three weeks later, successful bypasses fell from 37 to 4. The rejection rate for legitimate questions, which had climbed to 12% as the filters were tightened, came back down to 5% once the team redefined the scope of answers.