This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
AI chatbots including ChatGPT and Microsoft Bing have been documented generating false and defamatory claims about individuals, including fabricated sexual harassment allegations and criminal records. These systems often support these falsehoods with entirely invented citations and quotes from reputable news sources. The incidents highlight the propensity of large language models to confabulate information and the challenges in correcting such misinformation once it is generated.
A lawyer in California asked the AI chatbot ChatGPT to generate a list of legal scholars who had sexually harassed someone. The chatbot produced a false story of Professor Jonathan Turley sexually harassing a student on a class trip.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The underlying cause is the technical limitation of large language models to prevent hallucinations and verify the factual accuracy of their outputs.
The incident involves AI systems generating false and defamatory claims about individuals, supported by fabricated citations and quotes, leading to inaccurate beliefs.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The harm was caused by the AI system generating unexpected hallucinations after being deployed to the public.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
Commercial AI chatbots generated false, defamatory claims about individuals, fabricating realistic news citations. While causing reputational and psychological harm to specific persons, the incident represents a low national security threat, reflecting systemic epistemic challenges and model robustness limitations rather than coordinated warfare.
The AI systems (ChatGPT and Bing) function as conversational assistants that respond to user prompts, providing information and generating text under human direction and oversight.
Limited Risk: The report describes chatbots (ChatGPT and Bing) which are classified as limited risk under the EU AI Act, requiring transparency to ensure users know they are interacting with AI.
The systems involved (ChatGPT, Bing) are conversational AI models deployed as chatbots to answer user queries and generate text.
The reports provide detailed, first-hand accounts from the researchers and individuals affected by the AI hallucinations, including exact transcripts of the AI's fabricated outputs and quotes.
None
Technology
AI Hallucination and Confabulation
Models fabricate false facts, sources, and quotes with high confidence.
Lack of Fact-Verification Mechanisms
Models lack reliable mechanisms to verify the truth of generated statements.
Hard-Coded Filter Glitches
Crude filters mistakenly block benign names, breaking chat sessions.
Data Inputs
Unverified Training Data Sources
Models trained on scraped online content like Reddit without verification.
Feedback Loop of Misinformation
AI models scrape other AI-generated errors from the web, repeating them.
Human Factors
Overreliance on Confident Outputs
Users assume authoritative-sounding responses are factual without checking.
Adversarial Prompting by Users
Users prompt models to generate false or defamatory content.
Process and Methods
No Output Verification Tools
No automated check to verify if quoted text matches actual sources.
Crude Name-Blocking Remediation
Instead of correcting facts, OpenAI implemented crude hard-coded name blocks.
Management
Prioritizing Deployment Over Safety
Deploying models to the public before implementing robust verification checks.
Inadequate Risk Assessment
Underestimating the reputational harm of fabricated quotes and citations.
Regulatory Environment
Unclear Section 230 Protections
Uncertainty over whether AI creators are shielded from liability for libel.
Lack of AI-Specific Regulation
Absence of legal frameworks governing chatbot factual accuracy and liability.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
