This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
This report documents the practice of law enforcement agencies using unreliable 'probe images'—such as celebrity lookalikes, forensic sketches, and digitally altered photos—in face recognition systems to identify criminal suspects. The report highlights how these practices, often referred to as 'garbage in, garbage out,' lead to misidentifications and wrongful arrests. It criticizes the lack of transparency and due process, noting that defendants are rarely informed about the role of face recognition in their identification.
When the facial recognition search for a CVS theft suspect's face returned no useful matches due to the surveillance footage being obscured and highly pixelated, a New York City police detective continued the face search using Woody Harrelson's face allegedly due to his resemblance to the suspect's face, eventually leading to the arrest of an unknown victim.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
Also classified as
The facial recognition systems fail to perform reliably when subjected to non-photographic inputs like forensic sketches, yielding correct matches only 4% to 7% of the time.
Law enforcement officers are unsafely deploying facial recognition systems in critical situations by inputting highly flawed data (sketches, celebrity photos) and overrelying on the results to make arrests without proper independent corroboration.
Human
Due to a decision or action made by humans
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk of wrongful arrest occurs due to an unexpected outcome (misidentification) from pursuing the goal of identifying suspects using flawed inputs.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
The widespread and unregulated use of flawed 'probe images' by law enforcement agencies in facial recognition systems has led to wrongful arrests and systemic civil rights violations. While it poses no kinetic or foreign intelligence threat, it represents a substantial challenge to societal stability, privacy, and due process within the domestic justice system.
The facial recognition systems (Cognitec, Rekognition, Vigilant) operate as Tools. They perform the specific, well-defined function of matching an input 'probe photo' against a database of images and returning a list of candidates based on direct input without autonomous planning or context retention.
High Risk: The report describes facial recognition systems used in law enforcement applications to identify criminal suspects, which is explicitly classified as High Risk under the EU AI Act.
The systems are explicitly facial recognition technologies used to identify suspects from databases.
The report is highly detailed, citing specific cases (like the Woody Harrelson incident), police records, scientific studies, and official recommendations. The roles of the AI systems and the human operators are clearly delineated.
One could argue the failure is entirely human (police misconduct) rather than an AI safety issue, but the interaction between human editing and algorithmic confidence scores makes it a socio-technical AI safety incident.
Technology
FR Algorithms Misuse
Algorithms are not designed to match forensic sketches or edited photos.
Lack of Input Validation
The system does not reject low-quality or heavily modified probe photos.
High Confidence for Edited Photos
Algorithms return high confidence scores on heavily modified/fabricated faces.
Data Inputs
Use of Celebrity Doppelgangers
Detectives submitted celebrity photos instead of actual suspect images.
Submission of Forensic Sketches
Forensic sketches were used as probe images despite extremely low accuracy.
Fabricated Facial Features
Photos were edited to add features from unrelated online search results.
Human Factors
Overreliance on FR System
Detectives assumed the FR system's output was highly reliable.
Cognitive Bias in Matching
Analysts confirmed matches based on subjective and flawed inputs.
Creative Probe Selection
Detectives used celebrity lookalikes when suspect photos returned no matches.
Process and Methods
Flawed Image Editing Practices
Detectives edited photos with tools like Clone Stamp, fabricating features.
Lack of Corroboration Standards
No clear guidance on what independent evidence is needed to back up a match.
Inadequate Human Review
Subsequent reviews were conducted against edited photos rather than originals.
Management
Prioritizing Leads Over Accuracy
Management allowed questionable inputs to generate fast investigative leads.
Inadequate Officer Training
Lack of training on limits of facial recognition and data manipulation.
Lack of System Accountability
Departments failed to track search metrics, accuracy, and resulting arrests.
Regulatory Environment
No Rules on Probe Photos
No legal rules exist governing what images police can submit to FR systems.
Lack of Defense Disclosure
Defense attorneys are routinely denied access to FR search details.
Absence of Federal Standards
Lack of federal standards or oversight on police facial recognition usage.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
