This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Following an early leak of the open-source model Stable Diffusion on 4chan, users utilized the system to generate non-consensual pornographic deepfakes of celebrities. Experts and researchers highlighted that the model's lack of immutable safety controls, combined with its accessibility, enables the creation of harmful, targeted content at scale, posing risks of harassment, extortion, and the exacerbation of existing illegal behaviors.
Stable Diffusion, an open-source image generation model by Stability AI, was reportedly leaked on 4chan prior to its release date, and was used by its users to generate pornographic deepfakes of celebrities.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
The leaked model was used on public forums like 4chan to generate and distribute explicit pornographic and sexually explicit content.
The incident primarily involves the creation of non-consensual pornographic deepfakes and targeted sexual imagery to humiliate, harass, or potentially blackmail specific individuals.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The primary risk and harm are caused by human users intentionally prompting the leaked, open-source model to generate non-consensual pornographic deepfakes.
Added categories peak at 3: Substantial (Psychological Harm, Inferred)
The leak and subsequent abuse of the open-source Stable Diffusion model to generate non-consensual celebrity deepfakes highlights the evolving threat of accessible synthetic media. While posing minimal direct threat to critical infrastructure or state sovereignty, the incident demonstrates how unchecked generative AI capabilities can scale societal harms, privacy violations, and targeted harassment.
Stable Diffusion operates as a tool that performs a specific, well-defined function (generating images from text prompts) based on direct user input without autonomous planning or context retention.
Limited Risk: The report describes an AI system used to generate synthetic content and deepfakes, which falls under transparency and disclosure obligations under the EU AI Act.
The primary function of the AI system in this incident was generating photorealistic synthetic images, which users exploited specifically to create non-consensual deepfake pornography.
The report provides clear, detailed accounts of the Stable Diffusion leak, the specific platform where it occurred (4chan), the nature of the generated content (celebrity deepfakes), and the societal risks identified by experts. The roles of the developer and the users are explicitly defined.
The incident could be viewed purely as a copyright or open-source licensing issue, but the primary focus of the reported harm is clearly on malicious use and targeted harassment.
Technology
Bypassable Safety Classifier
The built-in Safety Classifier is optional and can be easily disabled.
Unfettered Model Architecture
The model itself is not restricted at a technical level from generating porn.
Low Consumer Hardware Barriers
Model runs on cheap consumer GPUs, enabling offline, unmonitored generation.
Data Inputs
Unfiltered Training Data
Model was trained on datasets allowing generation of celebrity likenesses.
Scraped Social Media Photos
Personal photos can be used to condition the model for targeted deepfakes.
Lack of Consent for Training
Training data included celebrity likenesses without explicit consent.
Human Factors
Malicious User Intent
Bad actors on forums like 4chan actively seek to generate non-consensual porn.
Underestimating Harmful Uses
Developers underappreciated how the system could be used in unanticipated ways.
Harassment and Extortion
Trolls and blackmailers utilize deepfakes to target and exploit individuals.
Process and Methods
Lack of Post-Deployment Controls
Once released in the wild, API rate limits and safety controls are bypassed.
Early Model Leakage
The model leaked early on 4chan before official safety gates were finalized.
Ineffective License Enforcement
The open license prohibited abuse but lacked technical enforcement mechanisms.
Management
Open Source Release Strategy
Releasing the model openly removed the ability to monitor or block abuse.
Inadequate Risk Assessment
Management did not fully foresee the scale of automated blackmail attacks.
Lack of Pre-release Governance
Inadequate internal coordination before the model leaked on public forums.
Regulatory Environment
Absence of Deepfake Laws
Few laws specifically protect individuals against AI-generated pornography.
Uneven Platform Enforcement
Image hosts and platforms enforce policies against deepfakes inconsistently.
Lack of Global Standards
No global standards exist to regulate open-source generative AI deployment.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
