This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The NYPD used a facial recognition system to identify a suspect in an indecent exposure case, resulting in the wrongful arrest and two-day detention of an innocent man. Despite the system flagging the match as only a 'possible match' and the individual having exculpatory phone location data, the police proceeded with the arrest based on a subsequent eyewitness identification. The charges were eventually dismissed, highlighting the risks of relying on facial recognition technology in law enforcement investigations.
The NYPD's facial recognition system allegedly misidentified Trevis Williams as a suspect in a Union Square indecent exposure case. Despite reportedly notable physical differences and exculpatory phone data, Williams was arrested, jailed for more than two days, and charged. The case was later dismissed.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
Police investigators over-relied on the AI's 'possible match' output, using it to build a case and arrest the suspect without conducting basic corroborative checks.
The facial recognition system failed to perform reliably under real-world conditions, generating a false match from grainy, low-quality surveillance footage.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was initiated by a false match generated by the facial recognition AI system during active law enforcement operations.
Added categories peak at 2: Minor (Psychological Harm, Indirect)
A local law enforcement incident where NYPD's overreliance on a facial recognition 'possible match' led to the wrongful arrest of an innocent man. While representing a serious civil rights concern, the incident has minor national security implications and is manageable within standard legal and judicial procedures.
The facial recognition system acted as an Assistant by analyzing surveillance footage and suggesting statistically similar candidates from a mugshot database, which a human examiner then reviewed and selected, explicitly warning that the result was only a 'possible match' and required further human investigation.
High Risk: The report describes the NYPD's use of facial recognition technology for law enforcement purposes, specifically to identify a criminal suspect from surveillance footage, which is classified as a high-risk application under the EU AI Act.
The system's primary goal was to perform facial recognition on surveillance footage to identify a suspect from a database of mugshots.
The report from the New York Times is highly detailed, providing specific names, dates, physical descriptions, and the sequence of events leading to the wrongful arrest. The role of the facial recognition system is explicitly described, along with the human decisions that compounded the error.
One could argue the primary failure was human police negligence rather than the AI system itself, though the AI's false match was the initiating event.
Technology
FRT Performance Drop on Grainy Images
FRT accuracy drops significantly on grainy surveillance footage.
Statistical Similarity Mismatch
Algorithm matched faces based on contours despite physical differences.
Data Inputs
Low-Quality Surveillance Footage
Search used grainy, blurry images instead of controlled-condition photos.
Mug Shot Database Inclusion
Active mug shot database included individuals with dismissed cases.
Human Factors
Cognitive Bias in Photo Array
Victim selected the suspect due to suggestion and confirmation bias.
Investigator Confirmation Bias
Detectives accepted the match without verifying physical discrepancies.
Examiner Misidentification
Human examiner selected the wrong candidate from the FRT suggestions.
Process and Methods
No Independent Corroboration Rule
NYPD does not require extra evidence before putting FRT match in lineup.
Failure to Verify Alibi
Police did not check phone location data or contact Amazon to verify alibi.
Inadequate Physical Verification
No process to reconcile physical differences like height (5'6 vs 6'2).
Management
Failure to Track System Errors
NYPD uses FRT thousands of times but does not track successes or failures.
Overreliance on Algorithmic Leads
Management allowed arrests based on FRT leads without rigorous checks.
Regulatory Environment
Lack of FRT Use Regulations
Absence of strict statutory limits on police facial recognition use.
No Standardized Oversight
No external regulatory body auditing accuracy or usage of police FRT.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
