This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Meta's AI chatbot in Facebook Messenger was found to be hallucinating false sexual harassment allegations against various New York state lawmakers. The chatbot fabricated specific details, including non-existent investigations and professional consequences, when prompted with the names of officials. While Meta acknowledged the technology is new and prone to errors, the incident has prompted calls for greater accountability regarding AI-generated misinformation.
Meta's AI chatbot in Facebook Messenger falsely accused multiple state lawmakers of sexual harassment, fabricating incidents, investigations, and consequences that never occurred. These fabricated stories, discovered by City & State, sparked outrage among the affected lawmakers and raised concerns about the reliability of the chatbot. Meta acknowledged the errors and committed to ongoing improvements.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The incident stems from the AI system's technical limitation of hallucinating realistic-sounding but entirely fabricated facts.
The chatbot generated and spread completely fabricated and false allegations of sexual harassment against individuals, which constitutes false or misleading information.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk was caused by the Meta AI system hallucinating false information, which was an unexpected and unintended outcome of its deployment.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
Meta's AI chatbot fabricated detailed, false sexual harassment allegations against several New York state lawmakers. While causing personal distress and highlighting generative AI misinformation risks, the incident was an unintentional technical hallucination with negligible direct impact on national security.
The report describes Meta AI as a chatbot that users converse with to get information, acting as a conversational assistant that responds to user prompts.
Risk Level 3. Limited Risk: The system is a chatbot, which falls under limited risk and requires specific transparency obligations to ensure users are informed they are interacting with an AI.
The system is a general-purpose conversational chatbot deployed on Facebook Messenger to answer user queries and generate text.
The reports are consistent, detailed, and describe a well-documented phenomenon of LLM hallucination with direct quotes from affected lawmakers and Meta representatives.
None.
Technology
Generative AI Hallucinations
The underlying LLM inherently fabricates details without verification.
Platform-Dependent Behavior
Inconsistent safety behavior across mobile, desktop, and login states.
Data Inputs
Lack of Verified Source Links
The model generated text without grounding in verifiable online sources.
Sensitive Prompt Vulnerability
Prompts with names and sensitive terms easily triggered false narratives.
Human Factors
User Prompting with Sensitive Terms
Users combined politician names with sensitive words, triggering safety gaps.
Process and Methods
Inadequate Content Filtering
Lack of robust guardrails to block generation of defamatory content.
Ineffective Multi-Platform Testing
Testing failed to catch variations in desktop versus mobile app responses.
Management
Premature Technology Deployment
Deploying generative AI features before resolving known hallucination issues.
Reliance on User Disclaimers
Shifting the safety and verification burden to users via general warnings.
Regulatory Environment
Absence of Defamation Regulations
Current laws do not specifically address AI-generated defamatory content.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
