This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Scammers are manipulating AI search and overview engines, specifically Google AI Overviews and ChatGPT, by populating the web with fraudulent customer service numbers. These numbers are then surfaced by the AI systems as authoritative results, leading users to contact impostors. One consumer reported losing $768 after being misled by a fraudulent number surfaced by Google's AI Overviews, highlighting a new vector for classic customer service hotline scams.
Google's AI Overviews allegedly surfaced a fraudulent customer service number for Royal Caribbean. Consumer Alex Rivlin reportedly called the number, spoke with an impostor, and provided his credit card, later incurring $768 in fraudulent charges. The same number reportedly appeared in ChatGPT results and was linked to Disney and Princess cruises, purportedly showing how scammers are manipulating AI outputs to amplify longstanding hotline scams.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
Scammers manipulated the AI's data sources to facilitate a financial scam.
The AI systems inadvertently surfaced and spread a fraudulent customer service number as an authoritative answer, leading a user to believe it was legitimate and suffer financial loss.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by Google's AI Overviews and ChatGPT unintentionally surfacing fraudulent phone numbers post-deployment due to data manipulation by external scammers.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
Scammers successfully exploited Google AI Overviews and ChatGPT through data poisoning, leading the systems to autonomously surface fraudulent customer service numbers. While representing a novel evolution of SEO fraud that erodes trust in AI search utilities, the national security impact remains minor and is manageable under standard fraud prevention procedures.
The AI systems function as search assistants, summarizing web information to answer user queries. The report notes that they 'spit out information gleaned from the web' to assist users, but require users to verify the results.
Limited Risk: The systems involved are search assistants and chatbots (Google AI Overviews and ChatGPT) which generate content and interact with users, falling under transparency obligations of Risk Level 3.
The primary goal of Google AI Overviews and ChatGPT in this context is to answer user queries and search the web to provide synthesized answers.
The report provides a clear, first-hand account of the victim's experience, verified by search analysts who replicated the issue on both Google AI Overviews and ChatGPT. The mechanics of the scam and the AI's role are well-documented.
The incident could be viewed purely as a traditional SEO spam/scam issue, but the integration of these numbers into AI-generated 'answers' (AI Overviews) represents a distinct AI safety failure.
Technology
AI Overviews Hallucination
AI generated responses presented unverified phone numbers as authoritative.
Susceptibility to Manipulation
Search and AI models easily manipulated by SEO spam techniques.
Data Inputs
Unverified Web Scraping
AI models ingested fake contact numbers from unvetted forums and blogs.
Lack of Vetted Databases
Systems failed to cross-reference phone numbers with official databases.
Human Factors
Overreliance on AI Answers
Victim trusted Google AI Overviews as highly authoritative.
Convincing Social Engineering
Scammers provided persuasive explanations and waived fees over the phone.
Process and Methods
Inadequate Content Filtering
Search algorithms failed to detect and filter out coordinated scam spam.
Delayed Information Update
AI models take too long to update after abusive content is removed.
Management
Prioritizing Speed Over Safety
Google deployed AI Overviews for phone queries without adequate vetting.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
