This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Predictive policing and risk assessment algorithms, such as PredPol and COMPAS, have been widely adopted by U.S. law enforcement. Investigations reveal these tools often rely on biased historical data, leading to the disproportionate targeting of minority and low-income communities. Furthermore, surveillance technologies like IBM's video analytics have been developed using secret access to public camera footage, raising significant civil liberties and privacy concerns.
Predictive policing algorithms meant to aid law enforcement by predicting future crime show signs of biased output.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
PredPol left millions of sensitive crime predictions and location data exposed on an unsecured AWS cloud storage bucket, and IBM developed video analytics using secret access to public camera footage.
Judges and police departments over-relied on algorithmic risk scores and patrol maps to make critical sentencing and law enforcement decisions, despite the systems' inherent biases.
The COMPAS algorithm exhibits unequal performance across demographic groups, falsely flagging Black defendants as high-risk at twice the rate of White defendants, while PredPol's accuracy and targeting disproportionately burden minority communities due to biased training data.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The systemic racial and socioeconomic biases and disproportionate targeting were caused by the outputs of the deployed AI systems, which was an unexpected and unintentional outcome of pursuing crime prevention goals.
Added categories peak at 3: Substantial (Psychological Harm, Inferred)
The deployment of biased predictive policing and recidivism algorithms across the United States has led to systemic civil rights violations and over-policing of minority communities. While it does not pose an immediate kinetic or foreign threat, the widespread erosion of trust in judicial and law enforcement institutions represents a substantial threat to domestic societal stability and human rights.
The AI systems (PredPol and COMPAS) function as decision-support tools that augment human capabilities. They generate predictions and risk scores (suggestions) but require human action and decision-making (by police officers and judges) to be executed.
High Risk: The report describes AI systems used in law enforcement and judicial applications, which have significant implications for fundamental rights, safety, and public interests.
The primary system analyzed is PredPol, which is designed for predictive policing to forecast crime hot spots. The secondary system is COMPAS, which is designed for recidivism risk prediction.
The reports provide extensive, empirical, and peer-reviewed evidence detailing the systemic biases of PredPol and COMPAS, supported by data analysis of millions of predictions and court records. The findings are highly credible and well-documented.
Proponents argue the software merely reflects reported crime rates rather than introducing new bias, though the reports strongly refute this.
Technology
Feedback Loop Amplification
Algorithm targets areas with historical arrests, reinforcing existing patterns.
Inaccurate Risk Predictions
COMPAS had high false positive rates for Black defendants.
Mixing Unrelated Crime Types
Combining different crimes into single predictions reduced accuracy.
Data Inputs
Biased Historical Crime Data
System relies on past arrests reflecting systemic policing biases.
Use of Socioeconomic Proxies
Questionnaires use poverty and family history as proxies for crime risk.
Incomplete Crime Reporting
Only a fraction of actual crimes are reported, biasing input data.
Human Factors
Automation Bias by Judges
Judges rely heavily on risk scores to override plea deals.
Officer Discretion and Abuse
Officers used predictions to justify stops and trespass arrests.
Lack of Public Awareness
Local advocates and task forces were unaware of software deployment.
Process and Methods
Lack of Model Transparency
Proprietary algorithms prevent public or defense scrutiny of scores.
Inadequate Validation Studies
Jurisdictions deployed tools without rigorous local validation.
Unsecured Data Storage
PredPol left millions of predictions exposed on an open AWS bucket.
Management
Ignoring Bias Research
Management refused to update algorithms despite known racial disparities.
Prioritizing Sales Over Safety
Selling software to agencies without enforcing proper usage guidelines.
Lack of Client Training
Failed to ensure clients did not predict discouraged crime types.
Regulatory Environment
Absence of Algorithmic Standards
No federal or state regulations govern predictive policing accuracy.
Lack of Independent Oversight
No regulatory bodies audited the software for civil rights violations.
Lack of Disclosure Mandates
Defense attorneys are not routinely informed when AI led to arrests.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
