This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A ProPublica investigation revealed that the COMPAS recidivism risk assessment tool, used widely in the U.S. criminal justice system, demonstrated significant racial bias. The algorithm was found to disproportionately misclassify black defendants as high-risk and white defendants as low-risk, despite the developer's claims of accuracy. The proprietary nature of the software prevented independent audit, raising concerns about accountability and the automation of systemic inequality in sentencing and bail decisions.
An algorithm developed by Northpointe and used in the penal system is two times more likely to incorrectly label a black person as a high-risk re-offender and is two times more likely to incorrectly label a white person as low-risk for reoffense according to a ProPublica review.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.
Also classified as
The proprietary nature of the COMPAS algorithm prevented public audit and made it impossible for defendants to challenge the basis of their scores.
The COMPAS algorithm disproportionately misclassified black defendants as high-risk and white defendants as low-risk, leading to systemic discrimination and unequal outcomes.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The biased risk scores were generated by the COMPAS AI system post-deployment, representing an unintentional outcome of using historical arrest data.
Added categories peak at 3: Substantial (Psychological Harm, Inferred)
The COMPAS incident highlighted systemic racial bias in automated risk assessment tools used within the U.S. judicial system. While representing a significant civil rights and societal fairness concern, its direct national security impact remains minor, as it does not threaten critical infrastructure, sovereignty, or national defense and is managed through domestic legal frameworks.
The reports describe COMPAS as a decision-support tool or risk-assessment assistant used by judges to inform bail and sentencing decisions. It generates scores but does not autonomously sentence defendants, requiring human judges to make the final legal decisions.
High Risk: The system is used in law enforcement and administration of justice, specifically for risk assessments and sentencing, which has significant implications for fundamental rights and public interests.
The primary function of COMPAS is to predict the likelihood of a defendant reoffending (recidivism prediction) to assist judges in making bail and sentencing decisions (judicial decision support).
The reports provide extensive, detailed statistical data from ProPublica's investigation, academic studies, and legal cases (Loomis v. Wisconsin). The core mechanism of the bias and its real-world impact are well-documented and consistent across sources.
Some researchers argue the tool is statistically unbiased because it achieves predictive parity (scores mean the same probability of reoffending regardless of race), and that the disparate error rates are mathematically inevitable.
Technology
Opacity of Proprietary Algorithms
COMPAS is a trade secret, preventing public audit or defense scrutiny.
Inherent Mathematical Trade-offs
Impossible to satisfy all fairness definitions when base rates differ.
Low Predictive Accuracy
Recidivism predictions are only slightly better than a coin flip.
Data Inputs
Biased Historical Arrest Data
Arrest data reflects systemic racial disparities and over-policing.
Proxy Variables for Protected Classes
Questions on family and neighborhood act as proxies for race and class.
Inaccurate Data Entry
Name and birthdate errors led to incorrect criminal record matching.
Human Factors
Automation Bias by Judges
Judges may over-rely on risk scores without understanding limitations.
Lack of Algorithmic Literacy
Users and defendants cannot interpret or challenge complex risk scores.
Societal Stereotypes in Research
Researchers utilized facial features to infer criminality without ethics.
Process and Methods
Arrests Used as Proxy for Crime
Using arrest records instead of actual convictions distorts risk scores.
Lack of Independent Validation
Many states use these tools without conducting validation processes.
Flawed Research Methodology
Criminality studies lacked criminology and physiology expert input.
Management
Prioritizing Proprietary Secrets
Vendor prioritized intellectual property over public accountability.
Inadequate Risk Assessments
Deploying tools before rigorously evaluating societal and racial impacts.
Neglecting Demographic Specifics
Failing to implement custom models for specific groups like women.
Regulatory Environment
Absence of Algorithmic Audits
Lack of federal mandates to audit machine learning tools for bias.
Weak Transparency Standards
No legal requirement to disclose source code of public decision tools.
Unresolved Legal Definitions
Courts struggle to define constitutional fairness in mathematical terms.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
