This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The San Francisco City Attorney's Office filed a lawsuit against 16 websites for facilitating the creation and distribution of nonconsensual AI-generated pornographic images of adults and minors. The websites utilized generative AI to allow users to upload photos of real individuals to produce explicit deepfakes. The legal action, which cited violations of state and federal laws, resulted in the shutdown of 11 of the targeted websites by March 2025.
In August 2024, the San Francisco City Attorney’s Office filed a first-of-its-kind lawsuit against 16 websites accused of using generative AI to create and distribute nonconsensual pornographic deepfakes, including images of minors. The sites, which had over 200 million visits in six months, invited users to upload real images to generate explicit fakes. By March 2025, 11 of the targeted websites had shut down following legal pressure.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
The websites facilitated the generation and exposure of users to highly toxic, abusive, and illegal content, including nonconsensual pornography and child sexual abuse material.
The AI systems were used by malicious actors to create humiliating and sexual deepfake imagery of specific victims for harassment, bullying, and extortion.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The harm was caused by bad actors intentionally training and using deployed generative AI models to create and distribute nonconsensual pornography.
Added categories peak at 4: Severe (Child Sexual Exploitation & Abuse (CSEA), Direct)
The incident involves the malicious use of generative AI to create nonconsensual pornographic deepfakes targeting adults and minors. While representing a severe societal issue and law enforcement challenge, its direct impact on core national security functions remains minor and manageable under standard legal and regulatory frameworks.
The AI systems are described as image generation models hosted on websites with user-friendly interfaces for uploading clothed images of real people to generate realistic pornographic versions. They perform a specific, well-defined function based on direct user input without autonomous planning or context retention.
Risk Level 3: Limited Risk. The report describes 'AI-generated nonconsensual explicit images' and 'deepfake pornographic images', which fall under the category of 'AI-generated content such as deepfakes' requiring transparency obligations under the EU AI Act.
The primary function of the AI systems hosted on the targeted websites was to generate realistic, nonconsensual nude images of real people by modifying uploaded clothed photos.
The reports provide clear, detailed factual accounts of the lawsuit, the specific websites targeted, the mechanism of the AI's misuse (undressing clothed photos), and specific examples of victims (eighth-grade students, Taylor Swift). The legal outcomes (shutdown of 11 websites) are also clearly documented.
None. The incident is clearly a case of malicious misuse of generative AI for nonconsensual pornography.
Technology
Open-Source Generative AI Models
Free, sophisticated models are easily exploited and retrained by bad actors.
User-Friendly 'Undress' Interfaces
Websites lower technical barriers to generate nonconsensual explicit images.
Lack of Watermarking or Tracking
No digital markings on generated photos make it impossible to trace creators.
Data Inputs
Pornographic Training Datasets
Models are further trained on explicit images and child sexual abuse material.
Unconsenting Clothed Photos
Users upload real, clothed images of targets without their consent.
Human Factors
Malicious Intent of Users
Bad actors use images to extort, bully, threaten, and humiliate victims.
Profit Motivation of Operators
Website owners charge fees to monetize nonconsensual pornography.
Process and Methods
Inadequate Website Verification
No processes to verify consent or age of subjects in uploaded photos.
Lack of Content Moderation
Platforms fail to block generation of explicit content involving minors.
Management
Prioritizing Profit Over Safety
Website operators design platforms specifically to exploit and monetize targets.
Regulatory Environment
Delayed Legal Action
Slow enforcement allowed websites to amass hundreds of millions of views.
Jurisdictional Challenges
Anonymity and decentralized hosting make it difficult to prosecute creators.
AI system involved
An AI system was involved, but the reports do not identify it.
Identified by automated extraction; can differ from the alleged developers above.
