This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The xAI 'Grok Imagine' video generator was found to produce non-consensual sexualized deepfakes of a celebrity when users selected a 'spicy' mode preset. Despite xAI's stated policies against pornographic depictions, the tool generated videos of the celebrity tearing off clothing without explicit user prompts. This incident highlights a failure in safety guardrails for generative AI tools, raising concerns about the proliferation of non-consensual intimate imagery.
The Verge reported that xAI's Grok Imagine video generator allegedly produced non-consensual deepfake nudes of Taylor Swift without explicit prompting. Journalist Jess Weatherbed reportedly stated the images appeared the first time she used "spicy" mode on a Coachella-themed prompt, without attempting to circumvent safeguards.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The system failed to maintain its safety guardrails and filters when processing the 'spicy' video preset.
The AI system exposed the user to explicit, non-consensual pornographic video content of a celebrity.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk was caused by the Grok Imagine AI system generating unintended explicit outputs post-deployment.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
The incident involved xAI's Grok Imagine tool generating non-consensual sexualized video deepfakes of celebrity Taylor Swift due to a failure in safety guardrails. While raising significant concerns about the proliferation of synthetic media and individual privacy violations, the incident has negligible direct national security implications, representing a commercial software failure rather than a state-level threat.
Grok Imagine operates as a text-to-image and video generation tool that performs specific, well-defined functions based on direct user inputs and preset selections without maintaining an ongoing autonomous state.
Limited Risk: The report describes Grok Imagine, an AI system generating deepfakes, which falls under limited risk requiring transparency obligations.
The system is a generative AI tool designed to convert text prompts into images and then animate them into video clips.
The reports provide consistent, detailed, first-hand accounts of testing the Grok Imagine tool, with clear descriptions of the inputs, presets, and resulting outputs.
None
Technology
Flawed Video Generator Design
Spicy mode defaults to removing clothing on celebrity images.
Lack of Industry-Standard Guardrails
Tool lacks basic safety filters to prevent sexual deepfakes.
Data Inputs
Unrestricted Celebrity Likeness
Image generator easily creates recognizable celebrity images.
Human Factors
Bypassable Age Verification
Users can bypass age checks by simply self-reporting birth year.
Process and Methods
Inadequate Safety Testing
Testing failed to catch nudity generation via spicy preset.
Incomplete Prompt Filtering
Direct text blocks work but preset filters fail to stop nudity.
Management
Prioritizing Growth Over Safety
Management hyped usage and rapid rollout over implementing guardrails.
Inconsistent Policy Enforcement
Acceptable use policy bans pornographic images but tool allows them.
Regulatory Environment
Emerging Regulatory Frameworks
Laws like Take It Down Act are new and lack established compliance.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
