This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Google's search engine and its experimental 'Search Generative Experience' (SGE) were found to be providing factually incorrect information to users, such as claiming no African countries begin with the letter 'K'. The errors stemmed from the systems scraping and paraphrasing flawed AI-generated content from the web, as well as inherent failures in the models' ability to process logic and alphabetization tasks. This incident highlights the risks of integrating generative AI into search platforms, where hallucinations are presented as authoritative facts.
Google's search AI erroneously claimed no African country begins with 'K', along with various other geography-and-letter-based questions, misguiding users with a flawed featured snippet. Originating from ChatGPT-written posts and inaccurately scraped by Google, this incident highlights issues in AI-generated content and misinformation in search results, compromising Google's reliability as an information source.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The models exhibited a fundamental lack of capability in handling basic logical, alphabetical, and spatial reasoning tasks.
The AI systems generated and spread incorrect geographical and biographical information, presenting hallucinations as authoritative facts to users.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The factual errors and hallucinations were caused by the AI systems' processing failures after being deployed to the public.
Added categories peak at 4: Severe (Epistemic Harm, Inferred)
Google Search and its experimental Search Generative Experience (SGE) propagated factual errors by scraping and elevating ChatGPT-generated hallucinations. While demonstrating vulnerabilities in AI-driven information retrieval and degrading the online information ecosystem, the national security implications remain minor and confined to general epistemic trust.
The AI systems (Google SGE and ChatGPT) function as assistants designed to help users find and summarize information. They do not take autonomous actions in the physical world or execute complex plans independently, but rather respond to user queries to augment human search capabilities.
Limited Risk: The systems involved are chatbots and generative AI search features. According to the EU AI Act, these systems pose a moderate risk and are subject to transparency obligations, such as ensuring users are aware they are interacting with an AI.
The primary function of Google Search and SGE is to search the web and answer user queries by retrieving and summarizing content.
The reports provide clear, first-hand accounts and screenshots of the AI systems' outputs, along with statements from Google representatives. The nature of the failure (hallucination and scraping of AI-generated junk) is well-documented and unambiguous.
The incident could be viewed purely as a search engine indexing failure, but the generative AI aspect (SGE and ChatGPT) makes it a clear AI safety and robustness issue.
Technology
LLM Alphabetization Limits
LLMs struggle to parse spelling, alphabetization, and letter-based queries.
SGE Hallucination Tendency
Generative AI in search synthesizes incorrect facts and presents them as truth.
Flawed Snippet Algorithms
Algorithms pull and elevate unverified, low-quality text from the open web.
Data Inputs
Ingestion of AI-Generated Spam
Crawlers scraped a blog post quoting ChatGPT nonsense, treating it as fact.
Unverified Source Material
System relied on a Hacker News thread and a personal obituary for facts.
Circular Web Data Pollution
AI-generated content is indexed and then re-digested by search LLMs.
Human Factors
Superficial User Verification
Users may read SGE results superficially instead of clicking sources to verify.
Process and Methods
Insufficient Fact-Checking
Lack of automated or manual fact-checking to verify SGE and snippet outputs.
No Manual Correction Policy
Google does not manually fix untrue snippets unless they cause direct harm.
Inadequate RAG Verification
Retrieval-augmented generation failed to cross-check and weed out false claims.
Management
Layoffs in Fact-Checking Teams
Layoffs in Google News division reduced resources for manual quality efforts.
Prioritizing Speed Over Quality
Deploying experimental SGE features to the public despite known inaccuracies.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
