This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A network of 17 TikTok accounts used AI text-to-speech technology, specifically ElevenLabs, to mass-produce and amplify conspiracy theories and false claims. These videos, which featured synthetic voices mimicking public figures, garnered over 336 million views. The incident highlights the ease with which AI tools can be weaponized to spread disinformation and the challenges platforms face in detecting and labeling synthetic media.
NewsGuard has identified 17 TikTok accounts that have been using AI-generated voices to advance and amplify conspiracy theories and false claims beginning in June 2023. By September 25, 2023, these accounts had amassed over 336 million views and over 14.5 million likes. Videos include baseless claims involving public figures such as Barack Obama, Oprah Winfrey, and Jamie Foxx.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to conduct large-scale disinformation campaigns, malicious surveillance, or targeted and sophisticated automated censorship and propaganda, with the aim of manipulating political processes, public opinion, and behavior.
Also classified as
The AI system was used to generate and spread highly realistic false claims and conspiracy theories about public figures.
The creators cloned the voices of specific public figures like Barack Obama to fabricate statements and manipulate public perception.
The incident involved a coordinated network of 17 TikTok accounts using AI-generated voices to conduct a large-scale disinformation campaign that garnered over 336 million views.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by human actors intentionally using ElevenLabs' voice cloning technology to spread disinformation after the tool was deployed.
Added categories peak at 4: Severe (Epistemic Harm, Direct)
A network of 17 TikTok accounts used ElevenLabs' voice cloning technology to run a massive disinformation campaign, generating over 336 million views with fake audio of public figures like Barack Obama. While the scale of exposure was substantial, the incident was managed through platform content moderation and did not directly compromise critical infrastructure, active government operations, or national security.
The report describes ElevenLabs as an AI text-to-speech tool where users upload recordings or input text to produce a digital copy or narration. It operates on direct input without autonomous planning or decision-making.
Limited Risk: The system is used to generate deepfakes and synthetic audio, which falls under Risk Level 3 due to transparency and disclosure requirements.
The primary function of ElevenLabs is to generate synthetic voices from text inputs (Voice Generation).
The reports from NYT and NewsGuard provide highly detailed, consistent, and verified information about the network of accounts, the specific AI tool used (ElevenLabs), the scale of views (336 million), and the specific false narratives spread.
None; this is a clear-cut case of malicious use of voice-cloning technology for disinformation.
Technology
Easy Voice Cloning Tools
ElevenLabs allows creating lifelike audio clones with a few clicks.
Flawed AI Voice Detectors
Detectors fail when background music or distortion is added to the audio.
Inherent Audio Flagging Challenges
Synthetic audio is uniquely difficult to flag consistently compared to video.
Data Inputs
Unrestricted Voice Uploads
Users could upload any voice recording to generate highly realistic clones.
Baseless Text-to-Speech Scripts
Conspiracy texts were fed into AI models to generate realistic narration.
Human Factors
Deceptive Creator Intent
Bad actors intentionally used AI to spread rumors and gain followers.
User Susceptibility to Audio Fakes
Viewers believed the AI voice claims, as audio is sticky in memory.
Process and Methods
Lack of Proactive AI Labeling
Creators failed to use TikTok's voluntary AI disclosure labels.
Inadequate Platform Detection
Social media platforms lacked robust automated tools to detect AI audio.
Management
Reactive Moderation Policies
Platforms acted only after NewsGuard flagged the viral misinformation.
Delayed Access Safeguards
ElevenLabs only restricted voice cloning after widespread public misuse.
Regulatory Environment
Lack of Industry Standards
No binding regulations exist to enforce labeling of synthetic media.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
