This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A nurse at St. Rose Dominican Hospital identified that an AI-generated sepsis alert was recommending an inappropriate IV fluid protocol for a dialysis patient. The nurse's clinical judgment, which recognized the risk of fluid overload, led to a physician's intervention that averted a potentially life-threatening complication. The incident highlights the risks of relying on automated clinical alerts that may conflict with patient-specific medical needs.
A nurse at St. Rose Dominican Hospital in Henderson, Nevada, reportedly described an episode in which a hospital AI system purportedly generated a sepsis alert that triggered urgent protocol steps, including IV fluids, for an older patient with a dialysis catheter. Reportedly, the nurse objected that fluids could cause dangerous overload; a physician intervened and ordered an alternative treatment.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The charge nurse insisted on administering the fluids despite clinical objections, demonstrating overreliance on the automated AI alert over human clinical judgment.
The AI system failed to perform robustly by generating an inappropriate and dangerous clinical recommendation for a patient with a dialysis catheter, reflecting a lack of capability in handling specific patient contexts.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk was initiated by an inappropriate sepsis alert generated by the deployed AI system, which did not account for the patient's specific clinical context of dialysis.
A clinical decision support AI at a US hospital generated an inappropriate sepsis treatment recommendation for a dialysis patient. The error was successfully intercepted by a nurse, preventing physical harm. The incident highlights localized clinical safety risks of AI integration in healthcare, with negligible implications for national security.
The AI system acts as an Assistant because it is designed to augment clinical capabilities by flagging sepsis risks to support care teams, rather than taking independent action. The report notes that the hospital views AI as 'one of the many tools that supports, not supersedes, the expertise and judgment of our care teams.'
Risk Level 2: High Risk. The system is an AI used in healthcare for clinical risk prediction, specifically a 'sepsis-prediction algorithm' used to flag life-threatening infections, which directly impacts patient safety and clinical decision-making.
The primary function of the sepsis-prediction algorithm is to analyze patient data to flag potential sepsis cases early, supporting clinical diagnosis and intervention.
The report provides a clear, detailed account of the specific clinical incident, the AI system involved, and the clinical reasoning used to avert harm. There is little ambiguity regarding the sequence of events or the role of the AI system.
None. The incident is clearly a case of a clinical decision support system generating an inappropriate recommendation that was successfully intercepted by human oversight.
Technology
Inaccurate Sepsis Algorithm
Epic sepsis model was found to be less accurate than marketed.
Vague BioButton Alerts
BioButton generated vague alerts without specific parameters, causing confusion.
Opaque Decision Making
AI models present scores without explaining the specific signals.
Data Inputs
Incomplete Patient Context
Algorithms lack access to physical signs observed directly by bedside nurses.
Biased Training Data
Some deployed algorithms were found to exhibit systemic racial bias.
Missing Clinical Parameters
BioButton tracked vitals but missed stroke indicators like slurred speech.
Human Factors
Alert Fatigue
High rates of false positives pulled nurses away from high-risk patients.
Automation Bias
Staff felt pressured to comply with AI alerts over physical clinical evidence.
Frontline Clinician Burnout
Staffing shortages and high workloads exacerbated the burden of false alerts.
Process and Methods
Rigid Protocol Integration
AI alerts triggered rigid protocols that demanded compliance over judgment.
Exclusion of Nurses in Design
Hospitals failed to consult frontline nursing staff during AI tool design.
Lack of Real-world Validation
Hospitals deployed unvalidated tools that fell short of real-world promises.
Management
Prioritizing Tech Over Staffing
Hospitals invested heavily in AI while bedside units remained understaffed.
Top-down Implementation
Management rolled out tools without clinician input, creating mistrust.
Inadequate Risk Assessment
Leadership failed to anticipate the operational risks of AI alert false alarms.
Regulatory Environment
No Centralized Gatekeeper
No single regulatory body validates AI tools before hospital deployment.
Unregulated Algorithmic Tools
Hospitals bypassed traditional safeguards when introducing new AI technologies.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
