This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Google's healthcare AI models, Med-Gemini and MedGemma, have been documented producing medical hallucinations and inconsistent diagnostic outputs. Experts warn that these errors, such as the invention of non-existent anatomical terms, pose significant risks in clinical settings due to potential automation bias and the difficulty of human verification in high-stakes medical environments.
Google’s Med‑Gemini healthcare AI reportedly produced the non‑existent term "basilar ganglia" in public launch materials, conflating two distinct brain structures. The error reportedly appeared in both a blog post and an arXiv preprint. Google is reported to have initially edited the blog without acknowledgment, later calling it a typo.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The AI models generated false medical terms and incorrect diagnoses that were published in research papers and promotional materials.
Experts highlight the severe risk of automation bias, where clinicians may over-rely on credible-sounding AI outputs and fail to catch critical medical errors.
The AI systems exhibited a lack of robustness by hallucinating non-existent medical terms and producing inconsistent diagnostic outputs based on minor prompt variations.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The medical errors and hallucinations were generated unintentionally by the AI systems during testing and pilot deployment phases.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
Google's Med-Gemini and MedGemma healthcare models demonstrated diagnostic errors and hallucinations during testing. While highlighting important safety concerns regarding automation bias in clinical settings, the incident remains a commercial software quality issue with negligible national security implications.
The AI models are designed to interpret scans, summarize data, and generate reports to assist clinicians, requiring human oversight and final decision-making.
High Risk: The AI systems are diagnostic tools used in healthcare, which is explicitly classified as a high-risk application under the EU AI Act.
The models are designed to interpret medical scans, generate radiology reports, and assist clinicians with diagnostic tasks.
The report provides clear, detailed accounts of specific errors made by Google's Med-Gemini and MedGemma models, supported by medical experts and researchers who documented the issues. The developer's response is also included, confirming the occurrence of the 'basilar ganglia' error.
Google downplayed the error as a simple misspelling or typo, whereas medical experts interpreted it as a dangerous hallucination demonstrating model limitations.
Technology
AI Model Hallucination
Model conflated basal ganglia and basilar artery into a nonexistent area.
Phrasing Sensitivity
MedGemma missed pathologies when queries were simplified or phrased differently.
Training Data Error Propagation
Model learned and repeated common mis-transcriptions present in training data.
Data Inputs
Inaccurate Training Labels
Training data contained mis-transcriptions that the model learned as correct.
Erroneous Clinical Notes
Incorrect assessments in notes are read and propagated further by the AI.
Human Factors
Automation Bias
Complacency leads clinicians to miss errors in generally accurate systems.
Reviewer Oversight
Over 50 authors and medical reviewers failed to catch the clinical error.
Process and Methods
Missing Confabulation Alerts
Systems lack real-time hallucination detection to warn clinicians of errors.
Flawed Review Workflows
Designing workflows where humans review highly hallucinated drafts is inefficient.
Non-Transparent Error Correction
Google quietly edited the blog post without updating the pre-print paper.
Management
Rapid Adoption of Immature Tech
Management pushed immature models into trusted tester and clinical pilots.
Downplaying Clinical Risks
Google downplayed a critical medical hallucination as a simple misspelling.
Regulatory Environment
Wild West Regulatory Stage
Healthcare AI lacks clear standards and high bars of evidence for deployment.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
