This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A widely used health-care algorithm developed by Optum was found to systematically disadvantage Black patients by using health-care costs as a proxy for medical need. Because historical systemic factors resulted in lower spending on Black patients, the algorithm incorrectly assigned them lower risk scores, effectively excluding them from programs providing extra medical support. The study highlighted how race-neutral metrics can perpetuate racial bias when applied to data reflecting existing societal inequalities.
Researchers analyzing an Optum algorithm used by a large academic hospital reportedly found evidence that it tended to under-predict the health care needs of Black patients, which in turn appeared to result in lower prioritization for extra care programs compared with white patients exhibiting similar health profiles.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
The algorithm's accuracy in predicting health needs was dependent on demographic group membership, performing significantly worse for Black patients by underestimating their severity of illness.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The discriminatory risk scores were generated by the deployed Optum algorithm, which unintentionally produced biased outputs due to using health-care costs as a proxy for medical needs.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
A widely used healthcare risk prediction algorithm was found to systematically discriminate against Black patients by using cost as a proxy for medical need. While representing a major civil rights and critical infrastructure resource allocation concern affecting millions, the national security impact remains minor as it was a non-malicious system bias remediated through standard research and corporate intervention.
The algorithm functions as a clinical decision support tool, generating risk scores to help clinicians and health systems identify patients for extra care programs, rather than taking independent action.
High Risk: The system is used in healthcare as a diagnostic or triaging tool to allocate medical care, which has significant implications for safety and fundamental rights.
The primary function of the Optum algorithm is to allocate healthcare resources by identifying and prioritizing patients who need extra care management. It also serves as a medical support tool to help clinicians gauge patient risk.
The reports provide consistent, detailed, and peer-reviewed scientific evidence from a study published in Science regarding the Optum algorithm's bias, its mathematical cause, and its real-world impact on patient care.
Some industry analysts questioned whether the primary failure was the tool itself or how health systems used it as a sole decision-maker rather than one of many data elements.
Technology
Cost Proxy for Health Needs
Algorithm used healthcare costs to predict needs, ignoring clinical realities.
Opaque Neural Networks
Image models learned to detect race through unknown visual shortcuts.
Sepsis Prediction Failures
Sepsis detection tool failed to predict illness in 67 percent of patients.
Data Inputs
Biased Historical Cost Data
Historical spending data reflected systemic disparities in healthcare access.
Lack of Training Data Diversity
Approved AI tools lacked clear information on demographic training diversity.
Unlabeled Racial Cues in Scans
Medical scans contained hidden racial patterns exploited by AI shortcuts.
Human Factors
Lack of Designer Diversity
Lack of diversity among designers led to unmitigated algorithmic harms.
Lack of Societal Context Training
Developers lacked training on social and historical contexts of health data.
Overreliance on Off-The-Shelf Tools
Hospitals used tools without supplementing them with clinician expertise.
Process and Methods
Proprietary Algorithm Opacity
Opaque software prevented researchers from auditing and detecting biases.
Insufficient Pre-Deployment Audits
Developers failed to routinely audit algorithms for bias before deployment.
Poor Clinician Procurement Input
Clinicians were not involved in purchasing decisions for clinical tools.
Management
Financial Incentive Prioritization
Systems focused on cost-containment incentives over equitable care delivery.
Neglecting Tool Limitations
Management viewed algorithmic scores as substitutes for doctor expertise.
Lack of Oversight Mechanisms
Organizations failed to establish mechanisms to detect algorithmic biases.
Regulatory Environment
Under-regulated Clinical Software
FDA did not require review of administrative and risk-prediction tools.
Non-mandatory Bias Testing
FDA only recommended rather than required testing for demographic bias.
Lack of Public Performance Disclosures
Companies were not mandated to publicly release demographic performance data.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
