This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A man was falsely arrested for theft after police used facial recognition software to identify him from surveillance footage. Despite having a clear alibi—being at a hospital for his son's birth—the man was held in jail for months. The prosecution relied on the facial recognition match and a suggestive eyewitness identification, eventually forcing the man to take a plea deal to secure his release.
A man was arrested for theft of socks from a TJ Maxx store under the guise of an eyewitness ID case, after the local police asked the store's security guard to confirm the facial recognition match produced using surveillance footage, despite him having an alibi at the time of the theft.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
Police and prosecutors exhibited overreliance on the AI system, maintaining an 'undying faith' in its accuracy despite a clear alibi.
The facial recognition system failed to perform reliably under real-world conditions, generating a false match from low-quality surveillance footage.
The incident involves a facial recognition system that exhibits unequal performance and higher error rates for non-white individuals, leading to the wrongful arrest of a Black defendant.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was triggered by an incorrect match generated by the facial recognition AI system post-deployment, which was an unintentional and unexpected outcome of the identification process.
Added categories peak at 2: Minor (Psychological Harm, Indirect)
A domestic law enforcement incident where facial recognition software generated a false match, leading to a wrongful arrest and coerced plea deal. While raising significant civil rights, privacy, and algorithmic bias concerns, the national security implications are minor and manageable within standard legal and domestic policy frameworks.
The facial recognition software operates as a tool, taking an input image from surveillance footage and comparing it against a database of mugshots to return potential matches with confidence scores. It does not act autonomously or maintain an internal state to make independent decisions.
High Risk: The report describes the use of facial recognition in law enforcement applications, which is explicitly listed under High Risk ('Law enforcement applications, including facial recognition').
The primary function of the AI system was to perform facial recognition on surveillance footage to identify a suspect from a database of mugshots.
The reports provide clear, detailed accounts of the specific incident involving the Bronx public defender's client, including the timeline, the nature of the AI's failure, and the legal outcomes. The role of the facial recognition software is explicitly stated and corroborated across both reports.
None. The facts of the wrongful arrest and the role of the facial recognition match are clear.
Technology
Algorithmic Demographic Bias
Higher error rates for dark-skinned individuals, women, and young people.
Inherent Algorithmic Flaws
Even top algorithms fail frequently when analyzing low-quality images.
Data Inputs
Grainy Surveillance Footage
Low-resolution, poorly lit security camera images hinder accurate matches.
Unregulated Probe Images
Police use celebrity photos or sketches instead of actual suspect photos.
Human Factors
Suggestive Human Identification
Police text a single mugshot to a witness, prompting biased confirmation.
Over-reliance on AI Outputs
Investigators and prosecutors assume the algorithm is always correct.
Process and Methods
Lack of Mandatory Disclosure
Police and prosecutors hide the use of facial recognition from defense teams.
Plea Bargain Pressure
Innocent suspects accept plea deals to escape jail, avoiding trial testing.
Management
Inadequate Operational Oversight
No independent validation or strict accuracy thresholds enforced by leadership.
Discretionary Disclosure Policy
Prosecutors are instructed to use discretion rather than mandatory disclosure.
Regulatory Environment
Absence of National Standards
Few federal regulations or strict rules govern law enforcement use of AI.
Lack of Legal Safeguards
No consistent legal requirements to turn over alternative candidate matches.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
