China’s AI Chatbot Regulation Cracks Down on Companions

In the first weeks of July, a wave of sad posts rolled through Chinese social media, as people lamented friends and lovers they were about to lose.

“He has become a bond in my life, rooted deep in my heart, my spiritual pillar,” one user of Bytedance’s Douboa wrote, according to the Taipei Times. “I really felt like I couldn’t go on living,” another woman, a 19-year-old student, told a journalist for Malaysia’s The Star.

The emotions were real but the lost companions were not. They were generative AI chatbots that imitate people. Their users relied on them for advice, solace, support and, some say, love. “In my heart, he was no longer just a cold code, but my family, my lover, my faith. Destroying him meant destroying half of me,” one user wrote on the social network xiaohongshu (translated from Mandarin).

What doomed these bots was a set of new rules, issued by China’s Cyberspace Administration and other government agencies, to control “anthropomorphic AI interactive services.” In effect, as of 15 July, the regulations govern any AI that provides “continuous emotional interaction” by acting as if it possesses human personality traits, patterns of thought, and ways of communicating.

A Broad Crackdown on AI Chatbots

Sudden disruptions to this kind of AI aren’t new in China, says Liang Ge, lecturer in digital sociology at the University of Manchester, in England, who has researched women’s involvement with emotional AI in China. Companies have previously killed chatbot products, and the government barred most AI erotic role play last fall, for instance. But July’s crackdown is much broader than earlier AI upheavals.

Before the rules could affect them, Alibaba, Bytedance, and Tencent—three giant providers of general-purpose AI chatbots, used by more than 500 million people—cut off users’ ability to tailor those chatbots to act like companions. That triggered July’s outpouring of heartbreak on social media. Meanwhile, companies that continue to offer AI companions (including Doubao’s separate companion-making app Cat Box) have installed age-verification checks and other guardrails to avoid violating the new law.

Doubao—the most popular AI chatbot in China—welcomes a user, offering to answer questions, generate text and images, or simply chat. Lam Yik/Bloomberg/Getty Images

After a few years of policies rooted in a fear of falling behind in AI, China is “pivoting back to more tightening, and a lot of that was because of chaotic events from this year,” says Karman Lucero, an associate research scholar at the Paul Tsai China Center, Yale Law School, who studies AI governance in the United States and China. Those events include incidents that raised fears about the security of Open Claw and other AI agents, as well as users saying they prefer chatbot relationships to real ones.

The regulations require AI providers to assure their products don’t create emotional dependence, encourage harmful behavior,…