This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
During the 2017 Notting Hill Carnival, the Metropolitan Police Service deployed an automated facial recognition system to identify suspects in the crowd. The trial resulted in a 98% false positive rate, with 35 false matches and one erroneous stop, highlighting significant concerns regarding the accuracy, privacy implications, and lack of oversight for biometric surveillance in public spaces.
The facial recognition trial by London’s Metropolitan Police Service at the Notting Hill Carnival reportedly performed poorly with a high rate of false positives.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The system scanned the biometric data of up to 2 million attendees without consent and matched them against unlawfully retained custody databases.
Police deployed and defended an immature technology, relying on its outputs to stop and question innocent citizens.
The primary failure was the facial recognition system's extremely high false positive rate (98%) and general lack of accuracy under real-world crowd conditions.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the deployment of an automated facial recognition system that generated unexpected false matches, leading to wrongful police stops.
The 2017 Metropolitan Police facial recognition trial at the Notting Hill Carnival demonstrated high failure rates, raising localized civil liberties and privacy concerns. However, the incident posed negligible threats to broader national security, critical infrastructure, or state sovereignty.
The AI system acts as an Assistant because it scans crowds and flags potential matches to augment police capabilities, but requires human officers to perform the final verification, intervention, and decision-making. As noted in the reports, 'additional checks and balances are in place to confirm identification following system alerts' and 'officers can quickly establish if the person has been correctly or incorrectly matched'.
High Risk (Level 2). The reports describe the deployment of 'Facial Recognition Technology' by the police for 'law enforcement applications, including facial recognition' to identify suspects in public crowds, which is explicitly classified as a high-risk application under the EU AI Act due to its significant implications for fundamental rights and public safety.
The system's primary function was to perform real-time facial recognition on crowds to identify individuals matching a police watch list.
The reports provide consistent, detailed accounts of the 2017 Notting Hill Carnival trial, including specific statistics on false positive rates (98%), the number of false matches (35), and the lack of legal/regulatory frameworks. Multiple sources (government, news, tech media) corroborate the core facts of the incident, leaving little ambiguity about the system's performance and the resulting civil rights concerns.
None
Technology
Immature Matching Algorithms
Poor match rates in crowded public spaces compared to controlled environments.
Lack of Demographic Bias Testing
Commercial software not tested for racial or demographic accuracy biases.
Data Inputs
Outdated Watch List Data
Suspect databases not updated, leading to stops of already-processed people.
Unlawful Custody Image Retention
Over 20 million custody images stored, including those of innocent people.
Lack of Standardized Database System
No single, shared system for storing and searching police-held images.
Human Factors
Automation Bias and Overtrust
Humans tend to trust algorithmic outputs without considering the consequences.
Intrusive Public Stops
Officers stopping and questioning citizens based solely on system alerts.
Process and Methods
Inadequate Trial Evaluation
Lack of proper evaluation, peer review, and publication of trial results.
No Mental Health Consultation
Watch lists compiled without consulting mental health professionals.
Management
Premature Technology Deployment
Police leadership deploying unproven systems in large public gatherings.
Defensive Posture on Failures
Management framing high false-positive rates as operational successes.
Regulatory Environment
Delayed National Biometrics Strategy
Government biometrics strategy delayed, leaving a policy vacuum.
Lack of Legislative Framework
No Parliament-approved laws or independent oversight for biometrics.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
