This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A 12-month pilot of an AI-based surveillance system in two South Australian aged care facilities resulted in over 12,000 false alerts. The high volume of false positives caused alert fatigue among staff, leading to a situation where they could not respond to every notification, ultimately resulting in at least one missed genuine resident fall.
Between March 2021 and March 2022, an AI-enabled video and audio monitoring system was trialed in two South Australian aged care facilities. According to an independent audit commissioned by South Australia Health, the system produced over 12,000 false alerts, overwhelming staff and contributing to at least one missed real incident. The report concluded the system did not reach accuracy levels acceptable to staff or management during the pilot.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The system was deployed for critical safety monitoring in an aged care setting before it was robust enough, leading to unsafe conditions when staff became overwhelmed.
The primary failure was the AI system's lack of capability and robustness, as it failed to perform reliably by generating 12,000 false alerts and missing a real fall.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the AI surveillance system generating excessive false alerts, which was an unintentional outcome of its deployment.
Added categories peak at 2: Minor (Psychological Harm, Indirect)
A pilot trial of an AI surveillance system in Australian aged care facilities failed due to excessive false alerts, leading to alert fatigue and one missed resident fall. This represents a localized technology failure with negligible national security implications.
The AI system operated independently to monitor residents, analyzing video and audio inputs in real-time to detect falls and screams, and automatically generated text alerts to an offsite monitoring center without human intervention.
High Risk (Level 2): The system is an AI-based safety and monitoring tool deployed in a residential care/healthcare setting, which has significant implications for the safety and physical well-being of vulnerable individuals.
The system was designed to monitor residents using cameras and microphones to automatically detect safety threats such as falls and screams.
The reports are highly consistent, drawing from an independent PwC audit of the trial. They provide specific details on the number of false alerts (12,000), the duration (12 months), the cost (AU $785,000), and the specific consequences (alert fatigue and one missed fall).
None. The failure of the system and its direct link to alert fatigue and the missed fall are clearly documented.
Technology
Inability to Distinguish Objects
AI struggled to separate inanimate objects like coats from actual people.
Overly Sensitive Audio Tuning
System triggered alerts on normal high ambient noise levels in facilities.
Misclassification of Crouch Motions
Staff crouching to help residents was misidentified as resident fall events.
Data Inputs
Poor Alignment of Action Patterns
Programmed movement and sound patterns did not match actual resident behavior.
Environmental Clutter and Occlusion
Multiple people or moving furniture blocked cameras or confused the AI model.
Human Factors
Severe Alert Fatigue
Staff stopped responding to alerts due to overwhelming number of false alarms.
Workforce Pressures
Existing staff shortages made responding to constant false alerts impossible.
Process and Methods
Underestimated Training Time
The trial underestimated the time required to train AI in an aged care context.
Inadequate Contextual Testing
Lack of comprehensive testing before launching the pilot across the sites.
Change in Technology Provider
Original monitoring company pulled out, leading to a botched rollout.
Management
Unrealistic Alert Expectations
SA Health anticipated only 10 false alerts per day, underestimating the risk.
Botched Rollout Management
Poor coordination and rollout of the trial as noted by the Health Minister.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
