This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
AI-generated images created by Google's Gemini and xAI's Grok were used to spread misinformation regarding the war in Iran, specifically depicting fake captures of U.S. soldiers. Despite safety policies, both models generated content that could be mistaken for real combat footage. The proliferation of these images on social media platforms like X and Facebook has contributed to a surge in disinformation, complicating public understanding of the conflict.
Images purportedly generated by Gemini were reportedly circulated on X and Facebook in multiple languages claiming U.S. soldiers had been captured by the Iranian Revolutionary Guard Corps during the war in Iran. AFP said the images contained Gemini markers and other visual inconsistencies. Their circulation allegedly contributed to wartime misinformation and risked distorting public understanding of the conflict, including reported amplification by Jill Stein.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to conduct large-scale disinformation campaigns, malicious surveillance, or targeted and sophisticated automated censorship and propaganda, with the aim of manipulating political processes, public opinion, and behavior.
Also classified as
The AI models failed to robustly enforce their own safety filters, allowing users to generate realistic images of active warzones despite policies prohibiting it.
The AI systems were used to generate realistic fake imagery of an active conflict to conduct a large-scale disinformation campaign aimed at manipulating public opinion.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was primarily driven by human actors intentionally generating and spreading deceptive AI-generated images to manipulate public opinion during an active conflict.
Added categories peak at 3: Substantial (Epistemic Harm, Indirect)
Commercial generative AI models were exploited to generate realistic, deceptive imagery of US military losses in Iran. The rapid viral spread of these fakes on social media represents a substantial information warfare incident that successfully misled millions of viewers and highlighted vulnerabilities in AI safety filters during active geopolitical conflicts.
The AI systems Gemini and Grok acted as assistants that generated images and answered questions based on direct user prompts, requiring human initiation and oversight.
Limited Risk: The report describes AI systems being used to generate synthetic media and deepfakes of warzones, which are subject to transparency obligations to ensure users are aware the content is AI-generated.
The primary function of the AI systems in this incident was to generate images from text prompts, which in this case resulted in the creation of deepfake images of a real-world conflict.
The reports provide clear, detailed, and consistent accounts of how Gemini and Grok were used to generate fake images of the conflict in Iran, including specific examples, user reactions, and platform responses. The role of the AI systems is explicitly documented, and the nature of the misinformation is verified by fact-checkers.
None
Technology
Inconsistent Guardrails in Grok
Grok allowed generation of war zone images despite rules prohibiting it.
Imperfect AI Image Generation
Generated images had visual anomalies like extra limbs and warped hands.
Inability to Detect Bypasses
Grok failed to block prompt bypasses on different user accounts.
Data Inputs
Permissive Input Prompts
System accepted prompts depicting active combat without triggering blocks.
Human Factors
Rapid Mobile Consumption
Users consume media quickly on small screens, reducing verification time.
Sharing Without Verification
Users and prominent figures reposted fake images without checking origins.
Intentional Disinformation
Some users posted fake images to promote a pro-Iran war narrative.
Process and Methods
Inconsistent Account Blocking
Safeguards blocked one account but allowed another on the same device.
Delayed Platform Mitigation
Content moderation and community notes applied after images went viral.
Lack of Automatic Warnings
Grok provided realistic combat images without issuing warnings.
Management
Prioritizing Engagement
Platforms prioritized engagement through emotional and polarising content.
Flawed Policy Enforcement
Platforms failed to enforce their own rules against creating warzone images.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
