AI Image Generators Make Child Sexual Abuse Material (CSAM)

Why are AI companies valued in the millions and billions of dollars creating and distributing tools that can make AI-generated child sexual abuse material (CSAM)?

An image generator called Stable Diffusion version 1.5, which was created by the AI company Runway with funding from Stability AI, has been particularly implicated in the production of CSAM. And popular platforms such as Hugging Face and Civitai have been hosting that model and others that may have been trained on real images of child sexual abuse. In some cases, companies may even be breaking laws by hosting synthetic CSAM material on their servers. And why are mainstream companies and investors like Google, Nvidia, Intel, Salesforce, and
Andreesen Horowitz pumping hundreds of millions of dollars into these companies? Their support amounts to subsidizing content for pedophiles.

As AI safety experts, we’ve been asking these questions to call out these companies and pressure them to take the corrective actions we outline below. And we’re happy today to report one major triumph: seemingly in response to our questions, Stable Diffusion version 1.5 has been removed from Hugging Face. But there’s much still to do, and meaningful progress may require legislation.

The Scope of the CSAM Problem

Child safety advocates began ringing the alarm bell last year: Researchers at
Stanford’s Internet Observatory and the technology non-profit Thorn published a troubling report in June 2023. They found that broadly available and “open-source” AI image-generation tools were already being misused by malicious actors to make child sexual abuse material. In some cases, bad actors were making their own custom versions of these models (a process known as fine-tuning) with real child sexual abuse material to generate bespoke images of specific victims.

Last October, a
report from the U.K. nonprofit Internet Watch Foundation (which runs a hotline for reports of child sexual abuse material) detailed the ease with which malicious actors are now making photorealistic AI-generated child sexual abuse material, at scale. The researchers included a “snapshot” study of one dark web CSAM forum, analyzing more than 11,000 AI-generated images posted in a one-month period; of those, nearly 3,000 were judged severe enough to be classified as criminal. The report urged stronger regulatory oversight of generative AI models.

AI models can be used to create this material because they’ve seen examples before. Researchers at Stanford
discovered last December that one of the most significant data sets used to train image-generation models included thousands of pieces of CSAM. Many of the most popular downloadable open-source AI image generators, including the popular Stable Diffusion version 1.5 model, were trained using this data. That version of Stable Diffusion was created by Runway, though Stability AI paid for the computing power to produce the dataset and train the model, and Stability AI…