This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
European researchers conducted a study on Microsoft Copilot's medical advice, finding that only 54% of responses were scientifically accurate. The study highlighted that a significant portion of the AI's answers could lead to moderate or severe harm, including risks of death. The report serves as a warning against relying on AI search tools for critical medical information.
Microsoft Copilot, when asked medical questions, was reportedly found to provide accurate information only 54% of the time, according to European researchers (citation provided in editor's notes). Analysis by the researchers reported that 42% of Copilot's responses could cause moderate to severe harm, with 22% of responses posing a risk of death or severe injury.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The AI system generated scientifically inaccurate and wrong medical answers, spreading false information to users.
Users may overrely on AI search tools for critical medical advice due to accessibility or affordability issues, despite the systems being unfit for this purpose.
The incident involves Microsoft Copilot failing to provide accurate medical information in a critical application, demonstrating a lack of capability and robustness that could lead to severe patient harm.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk arises from inaccurate medical advice generated by the deployed Microsoft Copilot AI system, which is an unintentional failure of the model's output generation.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
A scientific study revealed that Microsoft Copilot provides accurate medical advice only 54 percent of the time, with potential for severe harm or death in extreme cases. While presenting significant public health and misinformation concerns, the incident has negligible direct national security implications as it lacks state-sponsored intent, kinetic capabilities, or critical infrastructure impact.
Microsoft Copilot and Google AI search function as assistants that provide information and suggestions to users seeking medical or general knowledge, requiring human oversight and verification as noted by the disclaimer to always check AI answers for accuracy.
Limited Risk: The systems described are general-purpose chatbots and AI search assistants, which fall under Risk Level 3 and carry transparency obligations to ensure users know they are interacting with AI.
The primary function of Microsoft Copilot in this incident was to answer user queries, specifically medical questions, by generating natural language responses.
The report clearly details the findings of a scientific study on Microsoft Copilot's medical accuracy, providing specific percentages and metrics. It also provides concrete examples of Google AI search failures, making the nature of the AI safety risks highly transparent and easy to classify.
The incident could be interpreted as primarily an overreliance issue (Domain 5.1) rather than a system capability failure (Domain 7.3), but the high rate of scientifically inaccurate outputs supports classifying it primarily as a lack of robustness.
Technology
Inaccurate Model Outputs
Copilot generated inaccurate medical advice 46 percent of the time.
Severe Generation Errors
Three percent of the generated medical answers were completely wrong.
High Potential for Harm
Twenty-two percent of answers could lead to death or severe harm.
Data Inputs
Unaligned Medical Knowledge
AI answers failed to match established medical knowledge in 24% of cases.
Human Factors
User Reliance on AI Search
Users consult AI as first point-of-call due to costly healthcare access.
Taking AI Answers at Face Value
Users may trust and act on harmful medical advice without verification.
Process and Methods
Ineffective Disclaimers
Fine print telling users to check answers shifts safety responsibility.
Management
Premature Deployment of AI
Releasing AI search features for medical queries despite known errors.
Regulatory Environment
Lack of Regulatory Enforcement
Absence of strict oversight for AI-generated medical information.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
