This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Facial recognition technology deployed by retailers and police in the UK has resulted in multiple instances of mistaken identity. In one case, a customer was wrongfully accused of shoplifting and banned from stores, while in another, a man was detained by police after being incorrectly flagged as a wanted person. These incidents highlight concerns regarding the accuracy and potential for false positives in live facial recognition systems.
A facial-recognition software used by the British variety store Home Bargains is alleged to have misidentified "Sara" as a shoplifter, leading to staff searching her bag, escorting her from the premises, and banning her from the store. After, Facewatch is reported to have admitted its error to Sara. Facewatch is used by a number of different British stores.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
Store employees and police officers immediately acted on the AI's alerts to accuse and detain individuals without sufficient independent verification.
The primary failure is the AI systems' lack of accuracy and robustness, resulting in false positive matches that misidentified innocent individuals.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incidents were caused by false positive matches generated by deployed facial recognition AI systems, which was an unexpected outcome of their operation.
Added categories peak at 2: Minor (Psychological Harm, Direct)
The deployment of live facial recognition systems by UK police and retailers has resulted in instances of mistaken identity, wrongful detention, and store bans. While raising notable civil liberties, privacy, and domestic policing concerns, the incident has negligible direct national security implications and represents an established domestic regulatory challenge.
The AI systems operate as assistants by scanning faces and generating match alerts to augment human decision-making, requiring store staff or police officers to perform the final intervention.
High Risk: The reports describe the use of facial recognition in law enforcement and retail security, which falls under the High Risk category due to its significant implications for safety and fundamental rights.
The primary function of the systems deployed by both Facewatch and the Metropolitan Police is to perform real-time face recognition and match individuals against watchlists for identification.
The reports provide clear, detailed, and consistent accounts of two specific incidents of facial recognition failure, supported by statistics from the Metropolitan Police and statements from civil liberties groups. The role of the AI systems is explicitly stated, and the resulting harms are well-documented.
None. The incidents are clearly AI-driven misidentifications leading to wrongful accusations and detentions.
Technology
False Positive Match Rate
Algorithm misidentified innocent people due to matching error.
Family Resemblance Sensitivity
System struggled to distinguish between close family members' facial features.
Data Inputs
Inaccurate Watchlist Data
System compared passersby against watchlists containing flawed photos.
Human Factors
Automation Bias by Staff
Store employees blindly trusted the system alert without verifying identity.
Immediate Police Detention
Officers detained a citizen based on system alert without prior confirmation.
Process and Methods
Lack of Identity Verification
No robust verification process before accusing or detaining individuals.
Immediate Punitive Action
Immediate bans and searches implemented without secondary human checks.
Management
Premature Tech Deployment
Organizations deployed the system despite a 1 in 40 false alert rate.
Regulatory Environment
Absence of Specific Laws
Lack of legislation creates legal uncertainty and a 'Wild West' for tech use.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
