This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A trend on TikTok involves users creating AI-generated avatars using the Captions app to impersonate medical professionals, specifically gynecologists. These 'deepfake doctors' provide medically unfounded health advice and promote wellness products to millions of viewers. The deception has successfully misled thousands of users into requesting recipes and purchasing products, raising significant concerns about misinformation and potential health risks.
AI-generated avatars posing as OB-GYNs on TikTok, part of the so-called, crudely termed "Coochie Doctor" trend, have been falsely claiming years of experience while promoting dubious health advice. Many, including an avatar named "Violet," were created using the Captions app, which generates scripted AI influencers.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
The AI-generated avatars are used to spread unfounded and potentially dangerous medical advice to millions of social media users.
The incident involves the intentional creation of fake medical personas to deceive social media users and promote wellness products for financial and personal gain.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk is caused by human users who intentionally use the Captions app to generate deceptive medical personas and scripts to mislead the public for personal gain.
Added categories peak at 3: Substantial (Epistemic Harm, Indirect)
The incident involves the deployment of consumer-grade generative AI to create realistic deepfake medical professionals on TikTok for commercial fraud. While it poses public health and epistemic risks by spreading misinformation, the national security impact is minor, representing an evolved capability for consumer fraud rather than a coordinated state-sponsored threat.
The AI system, Captions, operates as a tool where users select an avatar, input a script, and the system deterministically generates a video based on those direct inputs without independent planning or decision-making. As Javon Ford demonstrated, users simply 'choose from a list of dozens of avatars, type or copy and paste a few sentences... and create a full TikTok video with a click'.
Limited Risk: The report describes AI-generated content such as deepfakes, which pose a moderate risk and require specific transparency obligations to ensure users are informed and aware of the AI's nature.
The primary function of the Captions app in this context is to generate realistic talking AI avatars (deepfake videos) from text scripts, including synthetic voice generation to match the avatar's lip movements.
The reports consistently describe the mechanics of the Captions app, the nature of the 'deepfake doctor' videos, and the public reaction. There is clear consensus across multiple sources (news.com.au, Daily Dot, Indy100, NY Post) regarding how the AI is being used and the specific medical claims being made.
The incident could be viewed primarily as a moderation failure by TikTok rather than an AI safety issue, but the central role of synthetic media generation makes the AI safety classification highly appropriate.
Technology
Realistic Generative AI Avatars
Captions app generates highly realistic human avatars like Violet.
Text-to-Speech Video Generation
Users can easily generate talking videos by typing scripts into the app.
Uncanny Visual Discrepancies
Avatars have stilted pauses, unblinking eyes, and audio-video lag.
Data Inputs
Unverified Script Input
Users can input any script, including false medical claims, without checks.
Cloned Human Likenesses
Companies clone real peoples likenesses to use as templates for avatars.
Human Factors
Lack of Viewer Discernment
Many viewers cannot easily distinguish AI avatars from real doctors.
Exploitation by Scammers
Malicious creators use AI to hawk wellness products and get comments.
Trust in Social Media Advice
Users readily believe medical advice seen on their TikTok feeds.
Process and Methods
Inadequate Platform Moderation
TikTok fails to filter out deepfake doctors spreading health misinformation.
Lack of AI Content Labeling
Platforms do not automatically label AI-generated videos as artificial.
Easy Video Production Process
One-click video creation allows creators to spam dubious content rapidly.
Management
Prioritizing Growth Over Safety
Captions app focuses on expanding user base rather than safety guardrails.
Inadequate Risk Assessment
App developers failed to foresee misuse of avatars for medical scams.
Regulatory Environment
Absence of AI Impersonation Laws
No strong legal framework exists to ban AI impersonation of professionals.
Lack of Medical Advice Oversight
Social media platforms lack regulation regarding medical advice dissemination.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
