EU’s AI Watermark Rules Challenge OpenAI’s TextGrain

The European Union now requires AI-generated text to be watermarked so that it can be identified as such. But tests on OpenAI’s new textGrain watermark, released in October and developed to comply with the new E.U. law, show that these watermarks often do not survive additional editing of the text or when it’s under 150 words. With no clear legal benchmark for how well watermarks should work, it’s uncertain whether textGrain will meet Europe’s transparency rules, which could cost the company significantly and make it difficult to detect AI-generated text as models grow more sophisticated.

AI models calculate the probabilities of different possible next “tokens” (the next word or word fragment it generates) based on the text already written, then select one and repeat that process until the response is complete. After the prompt “today the weather is…,” for example, says Xiang Li, coauthor of the textGrain technical report and a postdoctoral researcher at the University of Pennsylvania, the model might weigh “warm” or “cool” as possible next words.

A watermark uses a secret key to generate pseudorandom values—numbers produced by a mathematical process that influence which token the model suggests. This creates a statistical pattern in the text that a detector with the same secret key can look for, Li says.

What makes textGrain different from other watermarks is that it uses entropy, a measure of how unpredictable a model’s word choices are, as a resource that can be budgeted when embedding a watermark. Developers specify how much entropy or unpredictability they want to allow, and textGrain adjusts to that target.

This gives developers more control over the balance between embedding a detectable signal and preserving variation in the model’s output, Li says. “We can think of entropy as a resource, a kind of money.… In our approach, we can accurately determine how much of that resource we want to consume.”

Limits of AI Watermark Detection

The watermark is invisible to readers. But a positive result does not show how much a person contributed to a passage, and a negative result does not prove that a text was human-written.

OpenAI’s false-positive rate is 1 percent, meaning the detector is designed to wrongly flag about one in 100 unwatermarked passages. In tests on psychology questions from the ELI5 database, which gathers questions and answers from the r/explainlikeimfive subreddit, the model detected watermarks in about 80 percent of 200-token text (approximately 150 words) compared to 95 percent of 400-token text (approximately 300 words).

Li says this is a “basic limitation” of text-watermarking technologies and would be complicated to solve. Short passages may not contain enough watermark evidence for reliable detection, even with additional computation, he adds, while longer passages generally provide more.

Editing also weakens the signal. In an evaluation of 400-token passages, replacing just 10…