This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
An independent study of the Epic Sepsis Model (ESM) at Michigan Medicine revealed that the tool significantly underperformed compared to vendor claims. The study found the model missed 67% of sepsis cases and produced a high rate of false positives, leading to alert fatigue among clinicians. This incident highlights the risks of deploying proprietary, non-validated AI models in critical clinical settings.
Epic System's sepsis prediction algorithms was shown by investigators at the University of Michigan Hospital to have high rates of false positives and false negatives, allegedly delivering inaccurate and irrelevant information on patients, contrasting sharply with their published claims.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The ESM's mathematical formulas and training data were proprietary and shielded from public scrutiny by a corporate firewall.
The Epic Sepsis Model failed to perform robustly, missing 67% of sepsis cases and generating 88% false alarms in an independent validation study, demonstrating a severe lack of capability in a critical clinical application.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The underperformance and false alerts of the Epic Sepsis Model occurred post-deployment, representing an unintentional failure of the AI system to accurately predict sepsis.
Added categories peak at 2: Minor (Psychological Harm, Direct)
An independent study revealed that the widely deployed Epic Sepsis Model missed 67% of sepsis cases and caused severe alert fatigue due to an 88% false-alarm rate. While representing a significant patient safety and clinical quality concern within the critical infrastructure of healthcare, the national security implications remain minor and manageable within standard healthcare regulatory and operational frameworks.
The reports describe the ESM as a tool designed to 'help physicians diagnose and treat sepsis sooner' and 'help clinicians make timely decisions'. It does not take independent action but provides alerts to augment clinical decision-making.
High Risk: The system is an AI tool used in healthcare for diagnostic support ('medical AI systems' and 'tools that aid physicians in treatment and diagnosis'), which has significant implications for patient safety and health outcomes.
The Epic Sepsis Model is explicitly designed to analyze electronic health records to predict and help clinicians diagnose sepsis early.
The reports provide highly detailed, peer-reviewed scientific data from a JAMA Internal Medicine study validating the model's performance, including specific metrics like sensitivity (33%), specificity (83%), and false alarm rates (88%). The roles of the AI developer (Epic) and the validating institution (Michigan Medicine) are clearly defined, leaving very little ambiguity about the system's performance and the nature of the incident.
Epic argued that the poor performance was due to a lack of local calibration by the hospital rather than an inherent flaw in the model itself.
Technology
Proprietary Black Box Algorithms
Details inside the 'black boxes' are secret and independent tests are scarce.
High False Positive Rates
The model generated 88% false alarms, creating severe alert fatigue.
Algorithm Concept Drift
Nonadaptive algorithms perform worse over time due to concept drift.
Data Inputs
Biased Training Data
Algorithms trained on past spending data which reflects historical racial bias.
Geographic Data Bias
Training data relied on few states, causing poor performance elsewhere.
Inconsistent Data Terminology
Different hospitals used different terminology to catalog medical records.
Human Factors
Alert Fatigue
Clinicians faced thousands of false alerts, causing them to ignore warnings.
Uncritical Trust in AI
Belief in 'AI hype' led hospitals to trust algorithms without independent tests.
Failure to Calibrate
Hospitals failed to calibrate and tune the model for their local population.
Process and Methods
Lack of Local Validation
Hospitals deployed the tool without testing it in their clinical environment.
Varying Disease Definitions
Sepsis definitions varied, causing discrepancies between model and practice.
No Postmarket Surveillance
No routine monitoring was in place to track safety and performance over time.
Management
Business Goals Over Safety
Preserving market dominance clashed with the need for independent review.
Financial Conflicts of Interest
Vendor paid hospitals up to $1M to use algorithms, skewing safety priorities.
Lack of Developer Transparency
Proprietary models were shielded from scrutiny by a corporate firewall.
Regulatory Environment
Lack of FDA Oversight
Many clinical algorithmic tools were not required to be reviewed by the FDA.
No Mandatory Reporting
No central reporting system exists for patient harm caused by faulty AI.
No Required Bias Testing
Testing for racial and ethnic bias is suggested but not legally required.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
