This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
An algorithmic resource allocation system used in multiple US states to determine home care hours for disabled and elderly patients resulted in severe service cuts. The system failed to account for individual medical needs, leading to significant physical harm, neglect, and the threat of institutionalization for thousands of vulnerable individuals. Legal challenges in several states eventually led to the system being thrown out or modified due to its irrational and harmful decision-making.
A healthcare algorithm designed to equitably distribute caregiving resources drastically cut care hours for the disabled and elderly, leading to significant hardships and harm. Initially developed for fair resource allocation, the system ultimately faced legal challenges for its inability to accurately assess individual needs, resulting in reduced essential care and raising ethical concerns about AI in healthcare decision-making.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The algorithm replaced the professional judgments of nurses and social workers, and assessors blindly deferred to the computer's outputs without meaningful recourse for patients.
The algorithm failed to perform reliably or effectively under varying conditions, such as failing to factor in specific medical diagnoses like cerebral palsy, leading to wildly irrational and harmful care allocations.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The severe budget cuts and subsequent physical neglect were caused by the decisions of the deployed algorithmic resource allocation system, which produced unexpected harmful outcomes while pursuing the goal of resource distribution.
Added categories peak at 3: Substantial (Psychological Harm, Inferred)
An algorithmic resource allocation system used by several US states caused severe, arbitrary cuts to home care for disabled and elderly citizens, resulting in physical neglect and legal challenges. From a national security perspective, the impact is minor, representing a domestic administrative and civil rights issue that was successfully addressed through the US judicial system.
The algorithm operates as a deterministic tool. It takes direct inputs from a 286-question assessment and outputs a score that assigns a standard number of care hours, without maintaining an internal state, planning, or exhibiting autonomous goal-directed behavior.
High Risk: The system is used by public authorities to evaluate eligibility and allocate public assistance benefits and services, which has significant implications for safety and fundamental rights.
The primary function of the AI system was to allocate home care hours and budgets to disabled and elderly individuals based on assessment data.
The report provides clear, detailed accounts of the algorithm's deployment, its specific failures, the states involved, and the resulting physical and legal outcomes. The roles of the AI system and human actors are explicitly defined.
The incident could be viewed primarily as a policy and funding failure rather than an AI failure, though the algorithm's specific technical flaws and its role as the sole decision-maker support the AI safety classification.
Technology
Flawed Algorithmic Calculations
The algorithm failed to factor in critical conditions like cerebral palsy.
Binary Scoring Sensitivity
A single point score difference drastically changed the allocated care hours.
Data Inputs
Incorrect Assessor Inputs
Assessors made errors, like marking amputees as having no mobility issues.
Inflexible Questionnaire Data
The 286-question assessment failed to capture individual health subtleties.
Human Factors
Overreliance on Automation
Assessors blindly followed computer outputs without applying clinical judgment.
Process and Methods
Black Box Decision Making
Care decisions lacked transparency, making them extremely difficult to contest.
Inadequate Algorithmic Calibration
The system was designed to divide resources rather than assess actual needs.
Management
Prioritizing Cost Over Safety
States deployed crude algorithms to ration scarce care budgets arbitrarily.
Regulatory Environment
Lack of Algorithmic Oversight
No regulatory standards or oversight existed to govern healthcare algorithms.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
