This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
ElevenLabs' voice-cloning beta platform was exploited by users to generate and distribute non-consensual, hateful, and defamatory audio clips of celebrities and public figures on 4chan. The incident highlighted the risks of accessible generative audio technology, leading the company to implement stricter identity verification and safety safeguards.
Voices of celebrities and public figures were deepfaked using voice synthesis for malicious intents such as impersonation or defamation, and were shared on social platforms such as 4chan and Reddit.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The cloning of public figures' voices without consent to depict them saying offensive things is a form of targeted manipulation and harassment.
The incident involved the generation and distribution of highly toxic, racist, transphobic, and violent audio clips using cloned voices of celebrities on 4chan.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by human users intentionally exploiting ElevenLabs' beta platform to generate harmful and non-consensual deepfake audio clips after the tool was deployed.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
Anonymous users on 4chan exploited ElevenLabs' beta voice-cloning platform to generate and distribute non-consensual, hateful, and defamatory audio clips of celebrities and public figures, including US President Joe Biden. While the incident caused reputational and psychological harm, its national security impact remains minor as there was no evidence of coordinated state-sponsored information warfare or disruption to government functions.
The ElevenLabs voice cloning platform operates as a tool, performing the specific function of converting user-inputted text into speech using a cloned voice sample, without autonomous planning or context retention.
Limited Risk: The system generates deepfakes (AI-generated content), which under the EU AI Act are subject to specific transparency obligations to ensure users are informed of their synthetic nature.
The primary function of ElevenLabs' platform is to clone human voices and generate synthetic speech from text inputs.
The reports from multiple reputable sources consistently describe the same core event: ElevenLabs' beta platform being used by 4chan trolls to clone celebrity voices for toxic content, and the company's subsequent response. The details are clear and uncontradicted.
None
Technology
High-Fidelity Voice Cloning
Technology allows highly realistic voice clones from short audio samples.
Low Barrier to Entry
Easy-to-use web interface allowed rapid generation of deepfake audio.
Data Inputs
Minimal Audio Sample Requirements
Only one minute of clean audio was needed to clone a target voice.
Unverified Voice Ownership
No check to ensure users had rights or consent to the voice sample used.
Human Factors
Malicious Intent by Online Trolls
4chan users intentionally exploited the tool to generate hateful content.
Naive Assumptions of User Intent
Founders made naive assumptions about the goodwill of early adopters.
Process and Methods
Anonymous Free Tier Sign-Ups
Lack of identity or payment verification allowed anonymous abuse.
Absence of Pre-Release Safeguards
Beta launched without content filtering or voice verification tools.
Management
Feverish Rush to Market
Pressure to stand out in a noisy AI hype cycle led to cutting corners.
Inadequate Risk Assessment
Management failed to anticipate and prevent obvious misuse vectors.
Regulatory Environment
Lack of Solid Legislation
Flimsy or absent laws governing deepfake audio and non-consensual media.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
