This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Ousmane Bah, an 18-year-old student, was wrongfully arrested and faced multiple criminal charges across several states after Apple's security contractor and the NYPD used facial recognition technology to incorrectly link him to thefts committed by another individual. The misidentification was allegedly triggered by the real thief using Bah's lost learner's permit as identification. Despite evidence that the suspect did not resemble Bah, the AI-driven identification process led to repeated false accusations, causing significant personal, academic, and reputational harm to the plaintiff.
New York Police Department (NYPD)’s facial recognition system falsely connected a Black teenager to a series of thefts at Apple stores, which resulted in his wrongful attempted arrest.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The misidentification of two Black individuals who looked nothing alike highlights potential racial bias and misrepresentation in facial recognition performance.
The facial recognition system failed to perform robustly and accurately, misidentifying the plaintiff as the thief despite significant physical differences.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The harm resulted from the post-deployment operation of a facial recognition system that unintentionally misidentified the plaintiff due to incorrect database associations.
Added categories peak at 2: Minor (Psychological Harm, Indirect)
An 18-year-old student was wrongfully arrested due to a facial recognition misidentification propagated across private security and police databases. Although the incident highlights significant civil liberties and biometric privacy concerns, it has negligible implications for national security, critical infrastructure, or state sovereignty.
The facial recognition system acted as an Assistant by analyzing surveillance footage and suggesting matches to security personnel and law enforcement, who then made the final decisions to arrest and prosecute.
Risk Level 2: High Risk. The system is a facial recognition application used for law enforcement and security purposes, which is explicitly classified as high-risk under the EU AI Act.
The AI system was deployed to perform face recognition on surveillance footage and identify shoplifting suspects by matching them against database records.
The reports provide detailed, consistent accounts of the legal proceedings, the timeline of events, and the specific role of the facial recognition database in propagating the misidentification.
The incident could be viewed primarily as a case of human negligence and identity theft, with the AI system acting merely as a tool that scaled the error.
Technology
Facial Recognition Misidentification
System matched surveillance footage of impostor to Bah's incorrect record.
NYPD Database Search Confusion
Facial recognition search returned both the real thief and the victim.
Data Inputs
Flawed Identity Association
Impostor's photo was linked with Bah's name in the security database.
Non-Photo ID Vulnerability
Temporary learner's permit lacked a photo, allowing easy identity theft.
Human Factors
Failure to Verify Physical Matches
Staff ignored obvious height and physical differences between thief and ID.
Overreliance on System Alerts
Security and police accepted automated matches without manual verification.
Process and Methods
Flawed BOLO Creation Process
BOLO notices were generated linking victim's name to the thief's photo.
Negligent Evidence Retention
Apple and SIS failed to retain or deleted crucial store surveillance videos.
Management
Negligent Security Oversight
Apple and SIS failed to audit incorrect identity links in their databases.
Failure to Rectify Known Errors
Management continued using flawed data after mismatches were identified.
Regulatory Environment
Lack of Biometric Regulations
No strict legal standards governed private facial recognition database use.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
