This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Google disclosed to the Australian eSafety Commission that it received 258 global user reports of Gemini being used to generate deepfake terrorist content and 86 reports of it being used to create child abuse material between April 2023 and February 2024. While Google employs hash-matching to detect child abuse material, it acknowledged it does not currently use the same system for extremist content. The regulator described this as a world-first insight into AI misuse, highlighting the need for improved safety guardrails.
Google reported to Australia's eSafety Commission that it received 258 complaints globally about AI-generated deepfake terrorism content and 86 about child abuse material made with its Gemini AI. The regulator called this a "world-first insight" into AI misuse. While Google uses hash-matching to detect child abuse content, it lacks a similar system for extremist material.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The AI system lacked robust safeguards to prevent the generation of highly harmful and illegal content like extremist deepfakes.
The incident involves the generation of deepfake terrorist material and child sexual abuse material, which directly constitutes exposure to toxic, harmful, and illegal content as defined under sub-domain 1.2.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident involves human users intentionally utilizing the deployed Gemini AI system to generate prohibited and harmful content, representing a post-deployment misuse of the technology.
Added categories peak at 3: Substantial (Child Sexual Exploitation & Abuse (CSEA), Inferred)
Google reported that its Gemini AI was used to generate deepfake terrorist content and child exploitation material. While highlighting gaps in AI safety guardrails, particularly the lack of extremist content filtering, the incident represents a minor national security threat manageable through standard regulatory and law enforcement mechanisms.
The report describes Gemini as an AI program used by humans to generate specific content (deepfakes and text/images). It operates by responding to user prompts and augmenting human capabilities, which fits the Assistant behavior type.
Risk Level 3: Limited Risk. The report describes Gemini as an AI program used to generate deepfakes and other synthetic content, which falls under the transparency and disclosure obligations of Limited Risk systems.
Gemini was used to generate deepfake terrorist material and child abuse material, which falls under deepfake image and video generation.
The report clearly outlines the statistics of user complaints (258 for terrorism, 86 for CSAM) and Google's regulatory reporting to the Australian eSafety Commission. The roles of the AI system (Gemini) and the developer (Google) are explicitly stated, leaving little ambiguity about the nature of the incident.
None
Technology
Lack of Terrorist Hatch-Matching
Hatch-matching was not used to identify and remove terrorist material.
Inadequate Generative Guardrails
Gemini AI allowed generation of deepfake terrorism and abuse content.
Human Factors
Exploitation by Malicious Users
Users actively exploited Gemini to produce harmful and illegal content.
Process and Methods
Insufficient Safeguard Testing
Lack of effective testing of safeguards to prevent harmful generations.
Ineffective Complaint Verification
Google did not verify how many user complaints were policy violations.
Management
Inconsistent Safety Implementation
Management deployed safety tools for child abuse but not terrorism.
Regulatory Environment
Evolving Global AI Regulations
Regulators globally are still calling for better AI safety guardrails.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
