Why Are Large AI Models Being Red Teamed?

In February, OpenAI announced the arrival of Sora, a stunning “text-to-video” tool. Simply enter a prompt, and Sora generates a realistic video within seconds. But it wasn’t immediately available to the public. Some of the delay is because OpenAI reportedly has a set of experts called a red team who, the company has said, will probe the model to understand its capacity for deepfake videos, misinformation, bias, and hateful content.

Red teaming, while having proved useful for cybersecurity applications, is a military tool that was never intended for widespread adoption by the private sector.

“Done well, red teaming can identify and help address vulnerabilities in AI,” says Brian Chen, director of policy from the New York–based think tank Data & Society. “What it does not do is address the structural gap in regulating the technology in the public interest.”

What is red teaming?

The practice of red teaming derives its early origins from Sun Tzu’s military stratagem from The Art of War: “If you know the enemy and know yourself, you need not fear the result of a hundred battles.” The purpose of red-teaming exercises is to play the role of the adversary (the red team) and find hidden vulnerabilities in the defenses of the blue team (the defenders) who then think creatively about how to fix the gaps.

The practice originated in U.S. government and military circles during the 1960s as a way to anticipate threats from the Soviet Union. Today, it is mostly known as a trusted cybersecurity technique used to help protect computer networks, software, and proprietary data.

That’s the idea, at least. And in cybersecurity, where the role of hackers and the defenders are clear-cut, red teaming has a substantial track record. But how blue and red teams might be apportioned for AI—and what motivates the players in this whole exercise to ultimately act toward, ideally, furthering the public good—is unclear.

In a scenario where red teaming is being used to ostensibly help safeguard society from the potential harms of AI, who plays the blue and red teams? Is the blue team the developers and the red team hackers? Or is the red team the AI model? And who oversees the blue team?

Micah Zenko, author of Red Team: How to Succeed by Thinking Like the Enemy, says the concept of red teaming is not always well-defined and can be varied in its applications. He says AI red teamers should “proceed with caution: Be clear on reasoning, scope, intent, and learning outcomes. Be sure to pressure-test thinking and challenge assumptions.”

Zenko also reveals a glaring mismatch between red teaming and the pace of AI advancement. The whole point, he says, is to identify existing vulnerabilities and then fix them. “If the system being tested isn’t sufficiently static,” he says, “then we’re just chasing the past.”