This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
ChatGPT and InstructGPT models have been observed generating convincing but entirely fabricated information, including non-existent academic citations and false claims about the death of living individuals. These hallucinations, combined with the models' authoritative tone, pose risks for misinformation and potential harm in automated systems like recruitment or credit vetting. The incident highlights the challenges of model alignment and the potential for AI to produce harmful, unverified content.
When prompted about providing references, ChatGPT was reportedly generating non-existent but convincing-looking citations and links, which is also known as "hallucination".
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
Users relied on the chatbot to generate programming answers and academic references without verifying their correctness, leading to the pollution of platforms like Stack Overflow.
The model's tendency to hallucinate facts and generate fake references represents a lack of capability and robustness in critical information retrieval tasks.
The AI system generated false and misleading information, including fabricated obituaries of a living person and non-existent academic citations, which can lead to inaccurate beliefs.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The generation of fabricated information and fake obituaries was an unintended failure mode of the deployed ChatGPT model.
Added categories peak at 4: Severe (Epistemic Harm, Inferred)
The widespread public release of ChatGPT highlighted the system's tendency to generate highly convincing but entirely fabricated information, including fake obituaries and academic citations. While causing minor epistemic harm and individual distress, the incident represents a technological limitation rather than a coordinated national security threat.
The reports describe ChatGPT as a chatbot that answers questions, drafts contracts, and writes code, acting as an assistant to augment human capabilities but requiring human verification due to inaccuracies.
Limited Risk: The system is a chatbot generating text, which falls under transparency obligations to ensure users are aware they are interacting with an AI.
The primary function of ChatGPT is to act as an interactive conversational partner (Chatbot) and assist users in writing tasks (Writing Assistant).
The reports provide clear, first-hand accounts of the AI's hallucination behavior, backed by technical explanations from OpenAI's own documentation regarding InstructGPT and RLHF.
The incident could be viewed purely as a technical limitation (7.3) rather than misinformation (3.1), but the primary real-world impact described is the generation of false information.
Technology
Next-Word Prediction Objective
Model predicts likely words instead of verifying factual accuracy.
Authoritative Phrasing Bias
Model generates false information with a highly confident and formal tone.
Lack of Abstract Reasoning
Model struggles with abstract concepts, leading to logical errors.
Data Inputs
Uncurated Internet Training Data
Training on unverified web data leads to learning falsehoods and biases.
Ingestion of Personal Data
Processing personal data without consent leads to privacy violations.
Lack of Source Citation Data
Training data lacks structured citations, causing fabricated references.
Human Factors
User Over-reliance on Hype
Users accept outputs as absolute truth without verifying correctness.
Malicious User Manipulation
Users deliberately coax or jailbreak the model to output misinformation.
Lack of Verification by Users
Users publish AI outputs without checking facts or credentials.
Process and Methods
Inadequate Alignment Methods
RLHF fails to completely eliminate hallucinations and toxic outputs.
Inadequate Output Filtering
Lack of robust real-time filters allows false obituaries to be displayed.
Biased Labeler Preferences
RLHF training aligns with a narrow, English-speaking demographic.
Management
Hasty Public Deployment
Releasing unaligned models to the public driven by commercial pressure.
Poor Incident Response
Management failed to respond to reports of false death claims.
Regulatory Environment
Lack of AI Safety Regulations
No legally binding frameworks exist to enforce model safety standards.
Weak Data Privacy Enforcement
Regulators fail to prevent LLMs from harvesting and outputting personal data.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
