This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Multiple patients in Hyderabad and New York experienced serious health complications after following medical and dietary advice generated by ChatGPT. The incidents included a kidney transplant recipient losing her graft after stopping medication, a diabetic man developing severe electrolyte imbalances, and a man being hospitalized after consuming a toxic substance recommended by the AI. Clinicians are warning that the lack of clinical judgment and contextual understanding in AI tools makes them dangerous substitutes for professional medical care.
Reports from Hyderabad describe two alleged patient harms after individuals acted on purportedly ChatGPT-generated medical advice instead of clinician guidance. A kidney-transplant recipient reportedly discontinued prescribed post-transplant medications based on a chatbot suggestion and experienced graft failure. In a separate case, a man with diabetes allegedly developed severe hyponatremia after following a chatbot-advised zero-salt diet.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
Also classified as
The chatbot generated false medical claims that directly misled patients into taking harmful actions.
The AI system failed to perform safely by hallucinating toxic dietary advice (sodium bromide) and incorrect medical guidance.
The primary harm arose because patients placed inappropriate trust and reliance on a general-purpose chatbot for critical medical decisions, treating it as a substitute for professional clinical care.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The physical harm resulted from ChatGPT generating incorrect and dangerous medical and dietary advice post-deployment, which was an unintentional outcome of its text generation capabilities.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
Multiple individuals in India and the US experienced severe health complications after relying on medical advice from ChatGPT. While raising serious concerns regarding AI safety, consumer protection, and public health, the incident has negligible direct national security implications.
The report describes ChatGPT acting as an assistant that provides information, suggestions, and diet plans to users, but requires human validation and should not take independent clinical actions.
Limited Risk: The system involved is ChatGPT, which is a general-purpose chatbot. Under the EU AI Act, chatbots are classified as limited risk and are subject to transparency obligations to ensure users know they are interacting with an AI.
The AI system is ChatGPT, a general-purpose conversational chatbot designed to answer user prompts and generate text across various domains.
The reports provide clear, consistent, and detailed accounts of three specific medical incidents resulting from ChatGPT advice, including clinical details from senior medical professionals.
The incidents could be viewed purely as human error due to user negligence, but the AI's generation of highly specific, dangerous advice (like recommending sodium bromide) points to a system safety failure.
Technology
Lack of Clinical Judgment
AI lacks clinical judgment and context that a trained doctor brings.
Hallucination of Medical Facts
Chatbots hallucinate facts and cite outdated guidance with authority.
Inability to Detect Interactions
AI misses contraindications, drug interactions, and transplant protocols.
Data Inputs
Interpretation of Isolated Inputs
Chatbots interpret isolated inputs without looking at trend lines or charts.
Lack of Patient Longitudinal Data
AI lacks access to patient charts, medication history, and comorbidities.
Human Factors
Over-reliance on AI Advice
Educated and elderly patients rely on AI-generated advice over doctors.
Self-Discontinuation of Meds
Patients stop critical treatments based on chatbot outputs without checking.
Belief in AI Superiority
People start to believe that ChatGPT is superior to medical professionals.
Process and Methods
Lack of Clinic AI Guidance
Discharge summaries lack warnings against altering meds based on AI advice.
Absence of AI Triage Path
Clinics lack structured triage paths to review AI advice brought by patients.
No Medication Teach-Back Rules
Lack of teach-back processes to confirm patients follow prescribed regimens.
Management
Lack of Staff AI Training
Clinics lack structured training on responsible AI and verification safety.
No Team Policy on AI Use
Absence of clear staff policies for verifying clinician-facing AI outputs.
Regulatory Environment
Lack of Chatbot Accountability
AI tools have no duty of care, informed consent, or regulatory follow-up.
Lack of Standardized AI Governance
Absence of adherence to global ethics and governance standards for health AI.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
