This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
On April 4, 2024, X's AI chatbot Grok generated a false headline claiming that Iran had attacked Tel Aviv. This misinformation was automatically packaged and promoted by X's 'Explore' feature as a trending news story. The incident occurred because the AI relied on spam from verified accounts to generate its summary, demonstrating the risks of using automated AI for news curation without human oversight.
On April 4, 2024, X's AI chatbot Grok generated a false headline claiming "Iran Strikes Tel Aviv with Heavy Missiles," which was then promoted on X's trending news section. This misinformation, fueled by user spamming of fake news, falsely indicated a serious international conflict. The incident highlighted significant risks associated with relying on AI for content curation and demonstrated the potential for widespread dissemination of harmful misinformation.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
Grok failed to robustly filter out coordinated spam inputs, leading to a failure in its curation performance.
The Grok AI system generated and spread a false headline and narrative about a military strike, leading to the dissemination of misleading information to users.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The fake headline was generated by the Grok AI system after it was deployed, representing an unintended hallucination or error in its summarization goal.
Added categories peak at 4: Severe (Epistemic Harm, Indirect)
X's AI chatbot Grok autonomously generated and promoted a false headline claiming Iran had attacked Israel, triggered by automated ingestion of spam. While it caused no physical or governmental disruption, it highlights an evolved information security risk where automated, unverified AI news curation can rapidly scale geopolitical misinformation to millions of users.
Grok operated independently to ingest trending posts, make real-time decisions on how to summarize them, and generate a headline and narrative without human intervention or curation.
Limited Risk: Grok is an AI chatbot and generative text system, which falls under Limited Risk. The report notes that 'Grok is an early feature and can make mistakes' and includes a disclaimer, aligning with transparency obligations for chatbots.
The system's primary goal was to automatically curate and summarize trending topics on X to provide written context for users.
The report provides clear, direct evidence of the incident, including screenshots, timelines, and explanations of how the AI system generated the fake news. There is no significant ambiguity about the sequence of events.
None
Technology
Grok Generative Hallucinations
Grok AI chatbot fabricated a realistic headline based on false trend data.
Automated Trend Detection Algorithms
Algorithms flagged trending keywords without verifying the truth of the posts.
Data Inputs
Spam from Verified Accounts
Premium users spammed copy-and-paste misinformation that fed the AI.
Unverified Multimedia Inputs
Unverified videos of explosions were ingested as context for the trend.
Human Factors
User Trust in Official UI
Users trusted the AI headline because it was styled like a real news article.
Financial Incentives for Spammers
Paying users spread sensational fake news to monetize engagement.
Process and Methods
No Human-in-the-Loop Review
AI summaries were published directly to the homepage without human approval.
Inadequate AI Output Verification
System relied on a small disclaimer instead of verifying AI-generated text.
Management
Elimination of Human Curation Team
Management laid off human editors who previously verified trending topics.
Premature Feature Rollout
Management launched the updated Explore page despite known AI limitations.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
