This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
NarxCare and AvertD are predictive algorithms used in the US healthcare system to assess opioid misuse risk. Critics and researchers argue these tools lack transparency, peer-reviewed validation, and are prone to bias against vulnerable populations, leading to the unjust denial of pain medication and the stigmatization of patients.
NarxCare's overdose risk algorithm, lacking peer-reviewed validation, uses sensitive data like doctor visits, prescriptions, and possibly genetic information, leading to significant biases against women and Black patients. Factors like sexual abuse and criminal records exacerbate stigmas and disparities, often resulting in unjust denial of necessary pain medication. The newly approved AvertD genetic test shares similar issues, further complicating and potentially harming medical treatment decisions.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The algorithms incorporate biased data inputs, such as criminal records and confounded genetic markers, disproportionately impacting minority groups.
Clinicians and pharmacists blindly rely on the algorithmic scores to deny medication, ignoring warnings that the tool should not replace medical judgment.
The primary failure is the AI system's inability to reliably and accurately assess addiction risk, resulting in high false-positive rates that harm patients.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The harm is caused by the deployment of AI risk-scoring systems that unintentionally restrict access to healthcare for legitimate patients.
Added categories peak at 3: Substantial (Psychological Harm, Inferred)
The incident involves the domestic deployment of proprietary healthcare algorithms (NarxCare and AvertD) that generate biased opioid risk scores, leading to the denial of care. While causing individual distress and raising human rights concerns, the national security impact is minor and confined to domestic regulatory and societal policy domains.
The reports describe NarxCare and AvertD as clinical decision support tools designed to aid, not replace, medical decision-making by providing risk scores to doctors and pharmacists.
High Risk: The systems are AI tools used in healthcare to assess patient risk, which directly impacts their access to essential medical treatment and pain management.
The systems are explicitly designed to predict the risk of opioid abuse and assist medical professionals in diagnosing potential addiction risks.
The reports are highly detailed, from reputable investigative journalism sources, and cite specific academic studies, economic research, and multiple firsthand patient and physician accounts.
None that are plausible given the consistent evidence of algorithmic failure across all reports.
Technology
Proprietary Black-Box Algorithms
Opaque models prevent external validation and verification of risk scoring.
Low Algorithmic Precision
Low baseline rate of addiction leads to extremely high false-positive rates.
Genetic Ancestry Confounding
AvertD genetic testing is confounded by population stratification, causing bias.
Data Inputs
Confounded Veterinary Data
Animal prescriptions under owner names are flagged as human doctor-shopping.
Complex Medical History Bias
Legitimate care for cancer or chronic illness mirrors doctor-shopping patterns.
Biased Criminal Justice Data
Incorporating arrest records reinforces historical racial discrimination.
Human Factors
Automation Bias by Clinicians
Medical staff blindly follow scores instead of exercising clinical judgment.
Fear of Legal Prosecutions
Clinicians fear losing licenses or facing charges if they ignore high scores.
Stigma Against Addiction
Prejudice leads to immediate patient dismissal without clinical verification.
Process and Methods
Lack of Patient Redress
Patients have no clear pathway to correct erroneous data or appeal scores.
No Independent Validation
Algorithms are widely deployed without peer-reviewed safety validation.
Abrupt Care Termination
Clinics unilaterally dismiss patients without providing alternative pain care.
Management
Prioritizing Speed Over Accuracy
Platforms focus on rapid scoring rather than ensuring clinical safety.
Contradictory Marketing Claims
Developers market wide data integration but claim only PDMP data is used.
Lack of Developer Accountability
Companies deflect responsibility for clinical outcomes to individual doctors.
Regulatory Environment
Absence of FDA Approval Mandate
NarxCare operates as a clinical tool without requiring FDA safety clearance.
Contradictory FDA Approvals
FDA approved AvertD despite its advisory committee voting against it.
Mandatory PDMP Query Laws
State laws force clinicians to consult unvalidated risk scoring platforms.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
