This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A deepfake video featuring a prominent New Zealand endocrinologist was circulated on social media to promote a hemp gummy product. The video used AI to manipulate the doctor's likeness and voice, falsely advising diabetes patients to stop taking their prescribed medication, metformin. The incident highlights the growing risk of AI-generated impersonations of medical professionals to spread health misinformation and facilitate scams.
A video circulated on social media that appeared to feature University of Otago endocrinologist Sir Jim Mann allegedly promoting a hemp gummy product for diabetes patients and urging them to stop using metformin. The video was reportedly generated using AI and has been described as spreading manipulated visuals and speech presenting false medical advice.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
The deepfake video generated and spread highly dangerous medical misinformation, falsely advising diabetes patients to stop taking metformin.
The incident involves the unauthorized use of public likenesses (deepfake video and voice clones) to deceive consumers for financial gain through fake medical endorsements.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was initiated by human malicious actors who intentionally deployed AI deepfake tools post-deployment to perpetrate a commercial scam.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
The incident involves the use of AI-generated deepfakes and voice clones to impersonate medical professionals for commercial scams in New Zealand and the US. While posing risks to public health trust and individual consumers, the incident has minor direct national security implications and is manageable through standard law enforcement and regulatory procedures.
The AI systems were used as tools to generate synthetic media (deepfake video and voice clones) based on direct inputs of the doctors' likenesses, without exhibiting autonomous planning or agency.
Risk Level 3. Limited Risk: The report describes AI-generated content such as deepfakes ('AI is so clever that they could manipulate my mouth to be looking as if I was saying those words').
The primary method of the scam was generating a deepfake video of Dr. [name withheld], while a secondary occurrence involved cloning Dr. Brems's voice.
The reports clearly describe the nature of the deepfake scam, the platforms used, and the reactions of the affected individuals. The boundary rules clearly resolve the classification of public likeness exploitation under 4.3 rather than 2.1.
None
Technology
AI Deepfake Video Generation
AI was used to manipulate the doctor's mouth and clone his voice.
Voice Cloning Technology
AI cloned the podiatrist's voice to fake a medical recommendation.
Data Inputs
Publicly Available Likeness
Public media of the doctor was used to train the deepfake model.
Human Factors
Overreliance on Authority
People trusted the video because it featured a known medical specialist.
Lack of Digital Literacy
Viewers could not distinguish a highly realistic deepfake from a real clip.
Process and Methods
Weak Social Media Moderation
Facebook failed to detect and block the deceptive deepfake ad quickly.
Re-upload of Removed Content
The video occasionally resurfaced after initial removal.
Management
Inadequate Ad Screening
Platform management failed to screen high-risk medical ads effectively.
Regulatory Environment
Lack of Deepfake Regulation
Absence of strong legal frameworks to prevent AI impersonation of doctors.
AI system involved
An AI system was involved, but the reports do not identify it.
Identified by automated extraction; can differ from the alleged developers above.
