This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Multiple reports and studies, including a prominent 2016 ProPublica investigation, have highlighted concerns regarding the use of proprietary algorithmic tools like COMPAS and TrueAllele in the U.S. criminal justice system. These tools, used for recidivism risk assessment and DNA analysis, have been criticized for potential racial bias, lack of transparency due to trade secret protections, and performance that in some studies is no better than human judgment. The opacity of these systems has raised significant due process concerns for defendants unable to challenge the evidence used against them.
Correctional Offender Management Profiling for Alternative Sanctions (COMPAS), a recidivism risk-assessment algorithmic tool used in the judicial system to assess likelihood of defendants' recidivism, is found to be less accurate than random untrained human evaluators.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.
Also classified as
COMPAS was demonstrated to perform no better than untrained human volunteers or a simple two-variable linear predictor, showing a lack of capability.
The proprietary nature of COMPAS and TrueAllele prevents defendants and independent auditors from inspecting the source code, violating due process.
The COMPAS algorithm systematically treats black defendants unequally by falsely flagging them as high-risk at twice the rate of white defendants, leading to unfair sentencing and parole outcomes.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk of racial bias and inaccurate predictions stems directly from the outputs of the deployed AI systems, which was an unexpected and unintentional outcome of their design.
Added categories peak at 3: Substantial (Psychological Harm, Inferred)
The widespread deployment of proprietary, biased algorithmic tools like COMPAS in the US judicial system has led to systemic racial discrimination and due process violations. While not a kinetic or foreign threat, the scale of civil rights compromise and the erosion of public trust in core democratic institutions represent a substantial domestic societal stability concern.
The reports describe COMPAS and TrueAllele as tools that provide risk scores and DNA analysis to 'augment human judgement' and 'help judges make decisions' rather than taking independent action. The final decision authority remains with the human judges and parole boards, making the AI systems assistants.
High Risk: The EU AI Act classifies AI systems used in law enforcement, migration, asylum, and border control management, as well as administration of justice and democratic processes (such as risk assessment tools used by judicial authorities), as High Risk (Level 2).
The primary function of COMPAS is explicitly described as predicting the likelihood of a defendant reoffending (recidivism prediction) to support judicial decisions regarding bail, sentencing, and parole.
The reports provide extensive, consistent, and peer-reviewed scientific evidence (from Dartmouth researchers published in Science Advances) and investigative journalism (ProPublica) detailing the accuracy, bias, and transparency issues of COMPAS and TrueAllele. The legal and social implications are well-documented across multiple high-quality sources.
The developers argue that matching the 0.70 AUC standard is a validation of the tool's design rather than a failure, interpreting the results as proof of the model's validity.
Technology
Opaque Black-Box Models
Proprietary algorithms prevent courts and defendants from verifying accuracy.
Undetected Software Coding Errors
Bugs in software like STRmix and FST led to flawed outputs in court cases.
Ineffective Prediction Accuracy
COMPAS performed no better than untrained online survey takers.
Data Inputs
Biased Historical Arrest Data
Systemic bias in policing data led to racially disparate risk scores.
Proxy Variables for Race
Factors like prior convictions act as proxies, embedding racial bias.
Unrepresentative Training Data
Algorithms trained on non-Indigenous groups caused flawed risk assessments.
Human Factors
Cognitive Anchoring Bias
Judges and officers over-rely on risk scores without examining flaws.
Lack of Algorithmic Literacy
Judges did not understand how risk scores were mathematically computed.
Process and Methods
Group-Based Analytics Used
Use of group averages deprives defendants of individualized sentencing.
Lack of Source Code Audits
No independent adversarial testing was done on proprietary source codes.
Self-Validation of Software
Developers co-authored validation studies, creating conflict of interest.
Management
Commercial Profit Prioritization
Companies prioritized proprietary interests over transparency and justice.
Inadequate Risk Mitigation
Developers failed to proactively address bias and accuracy limitations.
Regulatory Environment
Trade Secret Protections
IP laws allowed companies to shield source code from defense scrutiny.
Lack of Regulatory Oversight
No standards require developers to prove algorithm accuracy before deployment.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
