This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
During the 2026 war in Iran, a significant volume of AI-generated images and videos were circulated on social media platforms, including X and Facebook. These fakes depicted non-existent wartime events, such as captured US soldiers and destroyed infrastructure, to influence public opinion and serve as propaganda. Investigations by AFP and The New York Times confirmed the use of tools like Google's Gemini and X's Grok to generate this content, which often bypassed safety guardrails and contributed to widespread information disorder.
Reports said that, in the early days of the war in Iran, purported AI-generated fakes showing nonexistent wartime scenes reached millions of viewers online. Their spread across social media and messaging apps allegedly distorted public perception of the conflict and contributed to information disorder.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to conduct large-scale disinformation campaigns, malicious surveillance, or targeted and sophisticated automated censorship and propaganda, with the aim of manipulating political processes, public opinion, and behavior.
Also classified as
The massive volume of AI-generated fakes flooded social media feeds, creating an alternate reality and eroding the shared consensus of real-world events.
AI systems like Grok failed to robustly enforce their own safety guardrails, allowing users to easily generate prohibited warzone imagery.
The primary risk involves state-aligned and malicious actors using AI systems to conduct large-scale propaganda and disinformation campaigns during an active war.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was driven by human actors intentionally generating and spreading AI-synthesized war propaganda to manipulate public opinion.
Added categories peak at 4: Severe (Epistemic Harm, Indirect)
During the 2026 war in Iran, malicious actors leveraged generative AI tools to conduct a large-scale information warfare campaign. By producing realistic fakes of captured US soldiers and destroyed assets, the campaign bypassed commercial guardrails to degrade public trust and consensus reality, highlighting the evolved national security threat of AI-driven wartime propaganda.
The AI systems (Grok, Gemini, Sora) operated as tools that performed the specific function of generating images and videos based on direct text prompts from users, without independent goal-directed planning or context retention.
Risk Level 3: Limited Risk. The AI systems described are used to generate synthetic content (deepfakes, AI-generated images/videos), which under the EU AI Act are subject to transparency obligations to ensure users are informed that the content is AI-generated.
The AI systems were primarily used to generate synthetic images and videos depicting realistic war scenes from text prompts.
The reports from reputable sources (AFP, NYT, iNews) provide consistent, detailed accounts of the proliferation of AI-generated imagery across multiple platforms, identifying specific AI tools (Grok, Gemini, Sora) and their watermarks. The evidence of guardrail bypass and the scale of dissemination is well-documented.
Some might interpret the incident primarily as a social media moderation failure rather than an AI safety failure, though the ease of generating realistic fakes via AI is central to the scale of the issue.
Technology
Flawed AI Guardrails
Grok failed to block prompts about active warzones despite internal rules.
Inconsistent Watermarking
AI watermarks were easily removed, obscured, or ignored by users.
Generative AI Imperfections
AI tools generated physical anomalies like extra limbs and malformed hands.
Data Inputs
Simple Text Prompts
Basic text instructions automatically generated highly detailed war imagery.
AI Enhancement of Real Footage
Real videos were edited with AI to exaggerate explosion sizes.
Human Factors
User Confirmation Bias
Users reposted fake images to support political views without verification.
Rapid Mobile Consumption
Users quickly viewed content on small screens, making fakes hard to spot.
Deliberate Disinformation Campaigns
Actors used AI-generated fakes as political propaganda to influence opinion.
Process and Methods
Weak Platform Moderation
Social media platforms failed to proactively filter or label AI fakes.
Bypassing Prompt Restrictions
Users easily bypassed Grok's warzone restriction from different accounts.
Delayed Fact-Checking
Community notes and manual sweeps occurred only after fakes went viral.
Management
Prioritizing Engagement Over Safety
Platforms prioritized emotional and polarising content to drive user views.
Inadequate Policy Enforcement
Platform management failed to enforce rules against generating warzone images.
Premature Tech Release
AI developers released advanced tools like Sora without robust abuse prevention.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
