This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
TikTok accidentally made an internal, unmoderated version of its 'Symphony Digital Avatars' tool accessible to the general public. This version lacked safety guardrails and mandatory AI-disclosure watermarks, enabling users to generate videos containing hate speech, dangerous health misinformation, and civic disinformation. TikTok subsequently removed the tool and characterized the incident as a technical error.
TikTok accidentally released an internal version of its AI digital avatar tool without safeguards, allowing users to generate videos where avatars could recite harmful content, including quotes from Hitler. The tool, meant for advertisers, was accessible to personal accounts and lacked the watermark indicating AI-generated content. TikTok has since removed the tool and acknowledged the problem.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The tool was used to generate civic disinformation telling people to vote on the wrong day and health misinformation about drinking bleach.
The incident involved a technical error where an internal testing version lacking standard safety guardrails and watermarks was exposed to the public.
The leaked AI tool allowed the generation of highly toxic content, including hate speech, white supremacist slogans, and dangerous instructions to drink bleach.
Human
Due to a decision or action made by humans
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by a human technical error that accidentally exposed an unmoderated internal testing version of an AI tool to the public.
TikTok accidentally leaked an unmoderated internal version of its Symphony Digital Avatars tool, allowing users to generate deepfakes containing hate speech, medical misinformation, and civic disinformation without watermarks. While the tool was quickly removed and no widespread dissemination occurred, the incident highlights the risks of accidental exposure of powerful synthetic media generation capabilities.
The AI system functions as a tool that takes direct text scripts as input and generates corresponding video and audio of digital avatars reciting the script without maintaining context or making autonomous decisions.
Risk Level 3. Limited Risk: The system is an AI video generator that creates synthetic content (deepfakes/digital avatars), which is subject to transparency obligations such as watermarking under the EU AI Act.
The tool is designed to generate realistic digital avatars of real people reciting user-provided scripts for social media advertising.
The reports provide clear, consistent details about the nature of the technical error, the specific content generated during testing, and TikTok's response. There is no major conflict between the sources.
None
Technology
No Generation Guardrails
The internal tool lacked active content filtering during video generation.
Missing AI Watermarks
Generated videos lacked watermarks identifying them as AI-created.
Data Inputs
Unfiltered Script Inputs
The system accepted any text script input without prior validation.
Human Factors
Accidental Link Exposure
An employee mistakenly posted a link to an internal testing tool.
Process and Methods
Weak Environment Isolation
Internal testing environments were accessible via public internet links.
Deficient Access Control
Personal accounts bypassed the Ads Manager restriction to use the tool.
Management
Inadequate Release Oversight
Lack of strict controls allowed internal testing tools to leak publicly.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
