This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
During a period of geopolitical tension, the AI chatbot Grok misclassified an authentic proof-of-life video of Israeli Prime Minister Benjamin Netanyahu as a deepfake. This output, which cited non-existent technical artifacts, was used to amplify existing disinformation campaigns claiming the Prime Minister was dead or injured. Experts noted that such AI-based verification tools are unreliable and prone to misinterpreting normal human behavior as synthetic anomalies.
During the Iran war, Grok reportedly told users that a café video posted by Israeli Prime Minister Benjamin Netanyahu was AI-generated or a deepfake, after the clip was released to counter rumors that he was dead or injured. Reuters verified the video's location and date and later reported that Netanyahu publicly dispelled the death rumors. Reports said Grok's answers amplified doubts about the clip.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The AI's false output was exploited by state-backed media to conduct a coordinated disinformation campaign during an active military conflict.
The AI chatbot failed to perform reliably under real-world verification conditions, demonstrating a lack of capability and robustness.
Grok generated false and misleading information by incorrectly labeling an authentic video of a political leader as a deepfake and fabricating technical anomalies.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the AI chatbot Grok generating false claims, which was an unexpected and unintended outcome of its operation.
Added categories peak at 4: Severe (Epistemic Harm, Inferred)
During a period of high geopolitical tension, the AI chatbot Grok falsely labeled an authentic proof-of-life video of Israeli Prime Minister Benjamin Netanyahu as a deepfake. This hallucination was quickly weaponized by Iranian state media to validate propaganda claiming the Prime Minister was dead or injured, demonstrating how commercial AI errors can be exploited for information warfare.
Grok operates as an AI chatbot designed to answer user queries and provide information, acting as an assistant to users trying to verify the video. The report notes that 'people turned to Grok... to help verify the authenticity of the footage,' showing it was used to augment human information-seeking.
Limited Risk: The system is an AI chatbot ('Grok --- X's AI chatbot') and generates synthetic text/content, which falls under Risk Level 3 due to transparency obligations for chatbots and AI-generated content.
Grok is explicitly described as X's AI chatbot, and users turned to it to help verify the authenticity of the footage, attempting to use it for fact-checking.
The reports provide clear, consistent details about Grok's specific outputs, the context of the geopolitical conflict, and the expert analysis of why the AI's verification failed. There is minimal ambiguity regarding the AI's role in generating the false deepfake claims.
One could argue the primary issue was the state-sponsored disinformation campaign itself, with the AI being a minor tool, but the AI's active fabrication of technical 'proof' makes it a distinct AI safety failure.
Technology
Unreliable AI Detectors
AI detectors falsely flagged authentic video of Netanyahu as digitally created.
Grok Chatbot Hallucinations
X's Grok chatbot falsely labeled the real cafe video as a deepfake.
Low Video Quality Distortions
Low-quality video compression distorted hands, creating a six-finger illusion.
Data Inputs
Probability-Based Detection
Detectors rely on statistical probability rather than absolute reality.
Low-Resolution Video Feeds
Low-quality footage led to visual artifacts used as false inputs.
Unverified Social Media Data
Grok used unverified social media claims as inputs for its analysis.
Human Factors
Confirmation Bias
Users wanted to believe the video was fake to support their narrative.
Overreliance on AI Tools
People trusted unverified AI detectors and Grok over real evidence.
Deliberate Disinformation
Pro-regime accounts spread fake news about Netanyahu's death.
Process and Methods
Lack of Fact-Checking Integration
X did not remove or correct Grok's false deepfake claims.
Flawed Verification Methods
AI detectors flag normal human anomalies as AI-generated artifacts.
Poor Content Moderation Response
Euronews contacted X about Grok's posts but received no response.
Management
Inadequate AI Risk Management
Platform management failed to mitigate real-time misinformation risks.
Failure to Monitor AI Chatbot
X management failed to prevent Grok from spreading misinformation.
Regulatory Environment
Absence of Chatbot Standards
No regulations governing the accuracy of real-time AI chatbot outputs.
Lack of Platform Accountability
No regulatory enforcement to compel platforms to correct AI errors.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
