This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A sexually explicit deepfake video of podcast host Bobbi Althoff went viral on X, reaching over 6.5 million views in less than 24 hours. The video, which superimposed Althoff's face onto an adult performer, was used by accounts for engagement farming and monetization. Althoff publicly debunked the video, confirming it was AI-generated and expressing distress over the graphic nature of the content. The incident highlights ongoing challenges with content moderation and the spread of nonconsensual deepfake pornography on social media platforms.
Nonconsensual deepfake pornography of Bobbi Althoff, which had been in circulation for six months, is reported to have suddenly gone viral on X, jumping from around 178,000 views to 6.5 million views over a matter of hours. In addition to the harm to Althoff, this incident also spotlights X's role in distributing AI-generated nonconsensual porn due to alleged lax moderation.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
The viral spread of the graphic, non-consensual pornographic video exposed millions of social media users to highly inappropriate and toxic content.
The incident involved the creation of humiliating and sexually explicit synthetic imagery targeting a specific victim for engagement farming and personal gain, which aligns with the definition of targeted manipulation and harassment under sub-domain 4.3.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The creation and distribution of the non-consensual deepfake video was an intentional act by human malicious actors using publicly available AI tools after their deployment.
Added categories peak at 4: Severe (Epistemic Harm, Inferred)
A viral non-consensual deepfake of a podcast host highlights the growing ease of generating highly realistic synthetic media and the challenges of platform content moderation. While causing severe individual psychological harm and violating personal rights, the incident has negligible direct national security implications.
The AI system was used as a passive tool by human creators to perform a specific, well-defined function (face-swapping) on direct input without exhibiting autonomous goal-directed behavior or planning.
Limited Risk: The report describes the creation and viral spread of an AI-generated deepfake video, which falls under the transparency obligations of Limited Risk systems under the EU AI Act.
The primary function of the AI system was to digitally superimpose a person's face onto another body in a video, which is the definition of deepfake video generation.
The reports provide consistent, detailed accounts of the incident, including direct quotes from the victim, researchers, and platform policies. The facts regarding the creation, spread, and impact of the deepfake are clear and uncontested.
None. The incident is clearly a case of non-consensual deepfake pornography used for harassment and engagement farming.
Technology
Advanced Face-Swapping AI
AI tools can seamlessly swap faces onto explicit videos with high realism.
Accessibility of Deepfake Apps
No technical skills needed, just downloading an app or clicking buttons.
Data Inputs
Stolen Original Video Footage
Using existing pornographic videos as base inputs for face-swapping.
Public Likeness Data
Using public social media images and videos of victims to train AI.
Human Factors
Financial Incentives for Users
Users share explicit content to farm engagement and monetize X accounts.
Lack of User Critical Thinking
Users believed the realistic deepfake was a genuine leak and shared it.
Malicious Intent of Creators
Creators share non-consensual porn to demean women and make money.
Process and Methods
Slow Content Moderation on X
Platform took over 24 hours to remove viral non-consensual explicit videos.
Ineffective Community Notes
Crowdsourced notes appear too late and do not stop the viral spread.
Monetization of Viral Engagement
Platform ad-share payouts incentivize the spread of controversial media.
Management
Gutting of Trust and Safety Teams
X management laid off moderation and curation staff, delaying responses.
Prioritization of Chaos over Safety
Leadership publicly dismissed content moderation, encouraging online chaos.
Regulatory Environment
Lack of Federal Legislation
No US federal laws exist to criminalize making or sharing deepfakes.
Absence of Platform Accountability
Platforms face no legal penalties for hosting non-consensual deepfakes.
AI system involved
An AI system was involved, but the reports do not identify it.
Identified by automated extraction; can differ from the alleged developers above.
