This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A TikTok user impersonated a banned public figure using AI-generated voice cloning to gain followers and promote a fraudulent 'wealth creation' course. The platform eventually banned the account after being alerted by media. The incident highlights the ease of using AI voice tools for impersonation and financial fraud.
A TikTok user was reportedly impersonating Andrew Tate, who was banned on the platform, by posting videos featuring an allegedly AI-generated audio of Tate's voice, which prompted his account ban.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
The AI-generated voice was used to spread the false narrative that a banned and imprisoned public figure had returned to the platform.
The incident involved using an AI-cloned voice to impersonate a public figure to deceive users into buying a fraudulent course for personal financial gain.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk was caused by a human actor intentionally using deployed voice cloning technology to commit financial fraud.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
An individual used consumer-grade AI voice cloning to impersonate a banned influencer on TikTok, successfully deceiving millions of viewers to promote a fraudulent course. While demonstrating the growing ease of synthetic media generation for financial fraud, the incident has negligible direct national security implications and was resolved through standard platform moderation.
The AI system was used as a tool to perform a specific, well-defined function of voice generation based on direct inputs without autonomous planning or decision-making.
Limited Risk: The report describes an AI system used to generate deepfakes (cloned audio), which falls under transparency obligations for AI-generated content.
The AI system was used to clone and generate the voice of a specific individual.
The report clearly details the impersonation scam, the platform used, the AI technology, and the financial motive. The boundary rules clearly resolve the classification between privacy and malicious fraud.
None
Technology
Accessible Voice Cloning
Startups like ElevenLabs make cloning voice models highly accessible.
Convincing AI Voice Synthesis
AI models generate highly realistic voices that easily deceive listeners.
Data Inputs
Publicly Available Audio
Abundant online audio files allow scammers to train voice cloning models.
Human Factors
User Susceptibility to Scams
Followers trust the AI voice and buy knock-off wealth courses.
Malicious Intent of Scammers
Actors exploit AI voice cloning to impersonate banned influencers.
Process and Methods
Reactive Content Moderation
TikTok only removed the impersonator after being alerted by reporters.
Ineffective Copycat Detection
Platform processes failed to proactively detect banned influencer copycats.
Management
Inadequate Moderation Oversight
Platform management failed to prevent evasion of bans by copycat accounts.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
