This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Scammers are using generative AI to create deepfake videos of Australian health experts and public figures to promote unapproved dietary supplements like 'Glyco Balance'. These fraudulent advertisements falsely claim that medical professionals endorse the products and that standard diabetes treatments are dangerous, leading to financial loss for consumers and potential health risks for patients who may stop taking prescribed medication. Despite efforts by platforms like Meta to remove the content, the volume and speed of these AI-generated scams remain a significant challenge for regulators and the public.
Scammers are reportedly harnessing AI-generated deepfakes of health experts and public figures in Australia in order to sell health supplements and give harmful health advice. Among the reported cases, deepfake videos are alleged to have falsely depicted Jonathan Shaw and Karl Stefanovic endorsing "Glyco Balance" for diabetes management, while Karl Kruszelnicki reportedly was falsely shown promoting blood pressure pills.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
The deepfakes spread dangerous medical misinformation, falsely claiming standard diabetes treatments are dangerous and promoting unproven supplements.
The incident involves malicious actors using generative AI to impersonate trusted medical professionals and public figures to scam individuals into purchasing unapproved supplements.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by human scammers who intentionally deployed generative AI tools post-deployment to create deceptive deepfakes for financial gain.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
Scammers utilized generative AI to create realistic deepfakes of Australian health experts to promote unapproved diabetes supplements. While posing minor physical risks to patients and eroding trust in public health, the incident remains a localized consumer fraud issue with negligible direct national security implications.
The report describes generative AI being used by criminals to make their work easier and more convincing. The AI functions as a tool to generate synthetic media based on direct inputs provided by the scammers.
Limited Risk: The report describes the use of AI-generated deepfakes to impersonate medical professionals, which falls under the transparency obligations of Limited Risk systems.
The AI system was primarily used to generate highly realistic deepfake videos of Professor Shaw and Karl Stefanovic, including cloned voices to make the scam convincing.
The report provides clear, detailed, and verified accounts of the deepfake scam, including a specific victim (Michael) and the targeted professional (Professor Shaw). The role of generative AI in creating the deepfakes is explicitly stated, and the resulting financial and epistemic harms are well-documented.
None
Technology
Generative AI Deepfakes
Generative AI allowed scammers to create highly realistic fake videos easily.
AI Voice Cloning
Scammers accurately cloned the voices of trusted medical professionals.
Limited Facial Recognition
Meta protective facial recognition is limited to select public figures.
Data Inputs
Stolen Video and Audio Data
Publicly available media was harvested to train AI deepfake models.
Fake Research Publications
Scammers uploaded fake reviews to platforms like ResearchGate to build trust.
Misleading Online Reviews
AI-generated testimonials were indexed online to create false credibility.
Human Factors
High Trust in Medical Experts
Patients naturally trust advice from well-known medical professionals.
Cognitive Fatigue
Users struggle to maintain constant vigilance when evaluating online ads.
Vulnerable Demographics
Elderly patients with worsening symptoms are highly susceptible to scams.
Process and Methods
Slow Moderation Response
Meta took nine days and required intellectual property claims to remove ads.
Whack-a-Mole Proliferation
Scammers rapidly post new fraudulent ads as soon as old ones are deleted.
Ineffective Auto-Detection
Automated systems failed to catch medical impersonation scams proactively.
Management
Safety Resource Underinvestment
Platforms prioritized rapid growth over investing in robust security tools.
Reactive Abuse Reporting
Management relied on slow user reporting rather than proactive prevention.
Inadequate Doctor Protections
Platform safety tools focused on celebrities rather than medical experts.
Regulatory Environment
Jurisdictional Gaps
Regulators struggle to police international cybercriminals hosting scams.
Inadequate Platform Liability
Lack of strict legal penalties for hosting fraudulent AI-generated ads.
Lax Identity Verification
Platforms do not mandate strict identity checks for publishing content.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
