This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
AI chatbots Gemini and Grok repeatedly provided false provenance information for an authentic photograph of a mass burial site in Minab, Iran, misidentifying it as disaster imagery from other countries. This behavior, characterized by authoritative but incorrect claims, complicates the work of human rights investigators and risks the denial of atrocities. The incident highlights broader concerns regarding the reliability of AI-generated summaries in news and information contexts.
Gemini and Grok reportedly gave false provenance claims about an authentic photo of graves prepared for victims of the Minab school strike, identifying it as disaster imagery from Turkey or Indonesia. Researchers verified the photo's location through satellite imagery and related visual evidence.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
Users are increasingly relying on AI summaries for critical news and information, treating probabilistic chatbots as factual 'truth boxes'.
The models failed to perform reliably, hallucinating facts and fabricating sources when faced with real-time information verification tasks.
The AI systems generated and spread authoritative but entirely false information about a real-world photograph, leading to inaccurate beliefs and undermining factual consensus.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The false information was generated by deployed AI systems (Gemini and Grok) behaving in an unexpected and inaccurate manner while attempting to answer user queries.
Added categories peak at 4: Severe (Epistemic Harm, Inferred)
Google Gemini and xAI Grok chatbots generated authoritative but false provenance information regarding an authentic photo of a mass burial site in Iran. This unintentional misinformation hindered human rights investigations and highlighted systemic risks of AI hallucinations eroding public trust in factual evidence during active conflicts, though it posed minimal direct threat to national security.
The report describes Gemini and Grok acting as conversational AI assistants and summaries tools that users turn to for answering questions and summarizing news, requiring human oversight to verify their outputs.
Limited Risk: The systems involved are conversational chatbots and AI-generated content generators, which are subject to specific transparency obligations under the EU AI Act.
Gemini and Grok are general-purpose conversational chatbots designed to answer user queries and summarize information, which they failed to do accurately in this incident.
The report provides clear, direct evidence of the AI systems' failures, including specific hallucinated responses, the real-world context of the photograph, and the direct impacts on researchers and victims' families. There is no significant ambiguity regarding the AI's role or the nature of the errors.
None
Technology
Probabilistic nature of LLMs
Models generate likely words rather than verifying factual truth.
Lack of visual analysis capability
AIs fail to analyze actual image pixels, relying on text associations.
Authoritative tone of false outputs
AIs present incorrect facts with high confidence and fake sources.
Data Inputs
Hallucinated source citations
AIs invent nonexistent links and news reports to back up claims.
Misassociation of historical data
AIs link current war images to old Covid or earthquake photos.
Human Factors
Overreliance on AI summaries
Users rely on AI summaries instead of checking primary news sources.
Perception of AI omniscience
Users trust AI as an objective 'truth box' free of human bias.
Increased skepticism of real media
Skepticism leads people to dismiss authentic atrocity images as fake.
Process and Methods
Ineffective chatbot correction loop
Prompting AI for corrections leads to further hallucinated errors.
Inadequate output verification
Platforms lack automated checks to verify AI-generated assertions.
Management
Premature public deployment
Tools were rolled out globally despite high documented error rates.
Lack of platform accountability
Companies ignore inquiries and rely on disclaimers to shift blame.
Regulatory Environment
Absence of liability for AI errors
Fine print disclaimers shield platforms from legal consequences.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
