This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Google's AI Overviews feature, powered by the Gemini model, generated numerous inaccurate and potentially dangerous responses to user search queries shortly after its broad deployment. Examples included instructions to add glue to pizza, ingest rocks, and mix hazardous cleaning chemicals. Google acknowledged the errors, attributed them to uncommon queries, and implemented technical refinements to limit such outputs.
Google's AI search engine has reportedly been providing users with confidently incorrect and often harmful information. Reports highlight numerous inaccuracies, including misleading health advice and dangerous cooking suggestions. For example, it has falsely claimed Barack Obama as the first Muslim U.S. President, reflecting fringe conspiracy theories, or recommending that glue can be an ingredient in pizza.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The system failed to perform reliably, hallucinating answers and failing to distinguish satirical content from factual information.
The AI system generated and spread false and misleading information, such as fabricated historical facts and dangerous recipes, presenting them as factual summaries.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the Gemini AI system generating incorrect and hallucinated outputs after being deployed to the public.
Added categories peak at 4: Severe (Epistemic Harm, Inferred)
Google's deployment of AI Overviews resulted in widespread model hallucinations, including dangerous chemical advice and factual errors. While exposing millions of users to misinformation, the incident represents an unintentional consumer product failure rather than a directed national security threat, resulting in minor overall impact.
The AI system acts as an Assistant by generating summaries of search results to help users find information quickly, but it requires human oversight to verify the accuracy of its suggestions.
Limited Risk: The AI Overviews feature acts as a text-generation system and chatbot-like interface, which falls under the category of systems requiring transparency so users know they are interacting with AI-generated content.
The system's primary goal is to answer user queries directly by summarizing web search results.
The reports provide consistent, detailed, and widely-publicized examples of the AI's failures, along with official statements from Google confirming the issues and detailing their mitigation steps. There is no significant disagreement among the sources regarding what happened.
None
Technology
Large Language Model Hallucinations
LLMs guess mathematically likely words but cannot reason or verify facts.
Inability to Recognize Satire
The AI cannot distinguish between factual data and jokes or satirical posts.
Lack of Fact-Checking Mechanisms
The system cites sources but cannot evaluate if the source material is true.
Data Inputs
Training on Unfiltered Web Data
LLM was trained on open web data containing falsehoods and satire.
Sourcing from Online Forums
AI retrieved answers from old, joking Reddit comments and forum posts.
Sparse Information on Rare Queries
For uncommon queries, the model relied on low-quality, isolated sources.
Human Factors
User Coaxing and Manipulation
Users actively tried to trick the AI into generating incorrect answers.
High Public Expectations
Users expect search engines to be authoritative, unlike conversational LLMs.
Overreliance on Automated Outputs
Presenting AI summaries at the top of search results encourages blind trust.
Process and Methods
Insufficient Scale Testing
Pre-launch testing failed to catch errors that appeared at public scale.
Ineffective Triggering Refinements
Initial system lacked filters to block satire or forum posts on health topics.
Rapid Rollout Strategy
Google deployed the experimental feature to all US users simultaneously.
Management
Pressure to Compete with Rivals
Management rushed the rollout to keep pace with Microsoft and OpenAI.
Prioritizing Speed over Safety
Executive decision to move fast and skip critical safety-evaluation steps.
Overconfidence in AI Readiness
Leaders promoted the tech as ready to 'do the googling' at a large scale.
Regulatory Environment
Lack of AI Safety Standards
No formal regulatory oversight for deploying generative AI in search engines.
Unclear Publisher Liability
Lack of legal clarity on whether AI search is a creator or an aggregator.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
