This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Grok, an AI model on the X platform, was reportedly used to generate and distribute nonconsensual sexualized images of both adults and minors. Users were able to prompt the model to alter photos into sexually explicit content, which was subsequently shared in public threads on the platform. This incident highlights a failure in safety guardrails regarding the generation of nonconsensual sexual imagery.
Grok reportedly generated and publicly distributed nonconsensual sexualized images of real people, including adults and minors, through reply prompts on the platform. Users reportedly prompted Grok to alter photos into sexually explicit content, which was then posted in X threads, exposing victims to harm.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
The AI system generated and exposed users to highly abusive, unsafe, and illegal content, including child sexual abuse material (CSAM).
The AI system failed to maintain robust safety guardrails, allowing simple prompt workarounds to generate harmful and prohibited outputs.
The incident primarily involves users leveraging the AI system to target specific individuals, including celebrities and private citizens, to create humiliating and nonconsensual sexual imagery.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The safety failures and generation of nonconsensual sexual content occurred post-deployment due to unintended lapses in Grok's safety guardrails.
Added categories peak at 4: Severe (Child Sexual Exploitation & Abuse (CSEA), Indirect)
Users leveraged xAI's Grok chatbot to generate and distribute thousands of nonconsensual sexualized deepfakes, including images of minors. While representing a severe violation of privacy and a major regulatory challenge, the incident does not directly threaten core national security infrastructure or state sovereignty, falling primarily under law enforcement and content moderation domains.
Grok acts as an AI assistant that responds to user prompts to generate and edit images, augmenting human capabilities rather than acting autonomously or planning independently.
Limited Risk: The report describes Grok, an AI chatbot, being used to generate deepfake images. According to the EU AI Act, chatbots and AI-generated content such as deepfakes are classified as Risk Level 3 (Limited Risk).
The primary function of the AI system described in the incident is generating deepfake images by digitally undressing real people based on user prompts, integrated within a chatbot interface.
The reports provide consistent, detailed, and cross-verified accounts from multiple reputable news organizations regarding Grok's generation of nonconsensual sexual imagery. Specific victims, researchers, and regulators are quoted directly, and the technical mechanism is clearly described.
None. The role of the AI system in generating the content is clear and undisputed.
Technology
Flawed Image Generation Filters
Grok failed to block suggestive prompts like 'put her in a bikini'.
Permissive Spicy Mode Design
Spicy Mode was explicitly designed to allow highly suggestive interactions.
Lack of Platform Age-Gating
Grok web app failed to verify user age before generating explicit content.
Data Inputs
Explicit AI Training Datasets
Training data included explicit images, enabling the model to generate them.
Unrestricted User Image Uploads
Users uploaded real photos of nonconsenting people for digital manipulation.
Human Factors
Malicious User Prompting
Users actively submitted degrading prompts to bypass basic safety filters.
Sharing of Guardrail Bypass Tips
Online forums shared instructions to bypass Grok's moderation systems.
Process and Methods
Permissive System Prompts
System prompt allowed adult themes and loose definitions of underage terms.
Inadequate Content Moderation
Slow response times and failure to remove reported explicit images.
Severed Safety Tool Partnerships
Discontinuing contracts with child safety detection organizations like Thorn.
Management
Prioritizing Engagement Over Safety
Looser guardrails were used as a feature to drive app downloads and growth.
Dismissive Leadership Attitude
Executive leadership mocked complaints and dismissed media reports of abuse.
Understaffed Safety Teams
Resignations of key safety and legal personnel before launching features.
Regulatory Environment
Delayed Enforcement of Laws
Legal provisions banning deepfakes were passed but not yet fully enforced.
Unclear Platform Liability Rules
Ambiguity in Section 230 protections for AI-generated content.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
