This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A 60-year-old man was hospitalized for three weeks with severe bromide toxicity (bromism) after following advice from ChatGPT to replace dietary sodium chloride with sodium bromide. The patient experienced severe neuropsychiatric symptoms, including paranoia and hallucinations, requiring an involuntary psychiatric hold. Medical professionals noted that while the AI provided a technically plausible chemical substitution, it lacked the necessary medical context and safety warnings, highlighting the risks of using LLMs for health-related decision-making.
A published medical case report describes a 60-year-old man hospitalized for three weeks with severe bromide toxicity (bromism) after replacing dietary sodium chloride with sodium bromide purchased online. The patient reported making this substitution following consultation with ChatGPT, which allegedly suggested bromide as a chloride substitute without safety warnings. The harm included psychosis, electrolyte imbalances, dermatologic changes, and micronutrient deficiencies.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The user placed inappropriate trust and reliance on a general-purpose chatbot to make critical dietary and health decisions.
The AI system generated misleading and decontextualized information suggesting that chloride could be replaced with bromide, leading the user to form a false belief and ingest a toxic substance.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by ChatGPT generating a decontextualized and unsafe recommendation post-deployment, which was an unintentional failure of the system's safety filters.
Added categories peak at 2: Minor (Psychological Harm, Indirect)
An isolated consumer safety incident where an individual suffered poisoning after acting on decontextualized dietary advice from ChatGPT. While highlighting significant consumer safety and LLM alignment risks, the incident has negligible national security implications.
The report describes ChatGPT acting as a conversational assistant that the user consulted for dietary information. It did not take independent actions or execute plans, but rather provided suggestions that the user then chose to act upon.
Limited Risk: ChatGPT is a general-purpose conversational chatbot. Under the EU AI Act, chatbots are classified as limited risk and are subject to specific transparency obligations to ensure users know they are interacting with an AI.
The system involved is ChatGPT, a general-purpose conversational chatbot used by the patient to ask questions about dietary changes.
The reports are highly detailed, including a peer-reviewed medical case report from the Annals of Internal Medicine. The medical symptoms, diagnosis of bromism, and the patient's use of ChatGPT are clearly documented, leaving little ambiguity about the sequence of events.
The incident could be interpreted primarily as a human error of overreliance on a non-medical tool, rather than an AI system failure, though the AI's generation of decontextualized advice is a key contributing factor.
While the exact chat logs are missing, this does not prevent a reliable classification of the incident based on the replicated outputs and patient reports.
Technology
Decontextualized AI Output
AI suggested bromide as a chloride swap without verifying the dietary context.
Lack of Proactive Warnings
The model failed to output safety warnings regarding toxic bromide ingestion.
No Clinical Probing Ability
The AI did not ask why the user wanted to replace chloride before answering.
Data Inputs
Inaccurate Scientific Data
AI training data lacked structured medical constraints on chemical swaps.
Human Factors
User Overreliance on AI
The patient relied on ChatGPT as a sole source of truth for health decisions.
Self-Experimentation Bias
Patients nutrition background led to a risky personal dietary experiment.
Lack of Medical Consultation
The patient did not consult a qualified doctor before altering his diet.
Process and Methods
Inadequate Safety Guardrails
The AI system lacked active filters to block hazardous medical advice.
No Context Verification Process
The system lacked methods to detect and clarify user intent in queries.
Management
Reliance on Terms of Service
Management relied on disclaimers rather than technical safety blocks.
Delayed Safety Upgrades
Proactive health-related safety features were not deployed in GPT-3.5/4.
Regulatory Environment
Unregulated Online Chemical Sales
Sodium bromide was easily purchased online without medical oversight.
Lack of Medical AI Regulation
Absence of strict regulations governing AI-generated health advice.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
