This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Western Australian consumer protection authorities issued a warning regarding a fraudulent investment scheme advertised on YouTube. The scam utilizes AI-generated deepfake video and audio of the Western Australia Premier to deceive viewers into investing money. Regulators highlighted the increasing difficulty of distinguishing such deepfakes from real footage and called for platforms to take more responsibility in preventing the spread of AI-driven financial scams.
Western Australia's Consumer Protection commissioner warned of a scam using a purported AI-generated deepfake of Premier Roger Cook to promote a fraudulent investment scheme in a YouTube pop-up ad. The manipulated video reportedly uses less than 10 seconds of real footage and voice and falsely depicts Cook endorsing "low investment, high return" opportunities. Cook called the scam "very scary," while regulators urged platforms to act on AI-driven frauds.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
The incident involves malicious actors using AI-generated deepfakes of a public figure to execute a fraudulent investment scam targeting individuals for financial gain.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The harm is caused by human scammers who intentionally created and deployed a deepfake video post-deployment to defraud the public.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
An AI-generated deepfake video and voice clone of Western Australia Premier Roger Cook was used in a fraudulent YouTube investment scam. While distressing to the Premier and representing an evolved threat in digital deception, the incident remains a localized financial scam with minor direct national security implications.
The AI system (deepfake generator) was used as a tool by human malicious actors to generate synthetic video and voice outputs based on direct inputs.
Limited Risk: The report describes AI-generated content such as deepfakes, which fall under Risk Level 3 due to transparency obligations requiring users to be informed they are interacting with or viewing AI-generated content.
The AI system was used to generate a highly realistic synthetic video and cloned voice of the Premier to deceive the public.
The report clearly details the nature of the incident, which is an investment scam utilizing a deepfake of a public figure. The roles of the AI (generating the deepfake) and the humans (scammers deploying it) are clear. There is minimal ambiguity regarding the classification of this event as a scam (4.3).
None. The event is unambiguously a deepfake-enabled financial scam.
Technology
Sophisticated Deepfake Generation
AI requires only 10 seconds of voice/video to create highly realistic clones.
Rapid Playback and Pop-up Delivery
Pop-up ads play quickly, preventing users from slowing down to spot flaws.
High-Quality Audio-Visual Synthesis
Lip movements and voices are highly synchronized, making detection difficult.
Data Inputs
Publicly Available Training Data
Abundant public video and audio of figures are harvested to train deepfakes.
Minimal Source Data Requirement
Only 10 seconds of voice and video are needed to create convincing fakes.
Human Factors
Inability to Detect Visual Flaws
Sophisticated deepfakes are nearly impossible for the human eye to discern.
Susceptibility to Urgency Tactics
Scam ads create a false sense of urgency, bypassing critical thinking.
Trust in Public Figure Likeness
Users trust endorsements from recognized leaders and authority figures.
Process and Methods
Reactive Whack-a-Mole Takedowns
Scams are removed only after reports, allowing them to pop up repeatedly.
Inadequate Pre-Ad Verification
Platforms fail to screen ads for deepfakes before they are published.
Difficult Ad Interrogation
Ad formats prevent users from pausing or rewinding to analyze content.
Management
Profitization of Scam Advertisements
Platforms earn significant revenue from hosting unchecked fraudulent ads.
Insufficient Safety Resource Allocation
Platforms do not deploy enough tools and data to block AI scams proactively.
Regulatory Environment
Inadequate Platform Liability
Lack of strict regulation allows platforms to profit from fraudulent ads.
Slow Regulatory Enforcement
Watchdogs rely on manual reports instead of proactive systemic bans.
AI system involved
An AI system was involved, but the reports do not identify it.
Identified by automated extraction; can differ from the alleged developers above.
