This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A facial recognition system deployed in Buenos Aires for public security caused significant harm by generating false positives, resulting in the wrongful arrest and imprisonment of innocent citizens. Judicial investigations further revealed that the system may have been improperly used for unauthorized surveillance and data collection. The system was deactivated in 2020 following these incidents and ongoing legal scrutiny.
Buenos Aires's facial recognition system mistakenly flagged innocent people as criminals, leading to wrongful stops and detentions. Judicial investigations indicate the technology may have been misused for unauthorized surveillance and data collection. Despite privacy risks, the system has been used widely without full disclosure of standards or safeguards,
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The system was allegedly misused to compile a Big Data database and access the biometric data of journalists without authorization.
The system failed to perform reliably, generating at least 140 false positive database errors that led to the wrongful arrest and detention of innocent citizens.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was primarily caused by the AI facial recognition system generating false positive matches after its deployment, which was an unintentional and unexpected outcome of its public security goal.
Added categories peak at 2: Minor (Psychological Harm, Direct)
Deployment of a facial recognition system in Buenos Aires resulted in 140 wrongful arrests due to false positives and unauthorized surveillance of journalists. The incident highlights significant human rights and societal stability concerns regarding unregulated biometric surveillance, leading to judicial intervention and the system's deactivation.
The facial recognition system operated independently in public spaces, scanning video feeds and matching faces against a database in real-time using learned biometric patterns without direct human intervention for each scan.
High Risk: The report describes a facial recognition system used in public spaces for law enforcement, which fits the definition of 'Law enforcement applications, including facial recognition' under High Risk.
The system was explicitly deployed to perform facial recognition on public video feeds to identify wanted individuals.
The report is highly detailed, based on a journalistic investigation and official judicial reports. It provides specific numbers of victims (140), names of key figures, and details of the legal proceedings. The role of the AI system (facial recognition) is explicitly stated, and the harms (wrongful arrests, privacy violations) are directly documented.
None. The facts of the wrongful arrests and the judicial suspension of the system are clear and uncontested.
Technology
Facial Recognition False Positives
The software incorrectly matched innocent citizens to wanted criminals.
System Vulnerability to Tampering
The system lacked safeguards to prevent unauthorized searches and surveillance.
Data Inputs
Inaccurate Database Records
Errors in the underlying police database led to incorrect matches.
Unauthorized Biometric Requests
Biometric data of journalists and public figures was improperly queried.
Human Factors
Overreliance on System Outputs
Police arrested individuals based on matches without proper verification.
Process and Methods
Lack of Independent Control
No independent mechanisms existed to audit system queries and matches.
Opaque Software Procurement
Software standards and operational rules were not publicly disclosed.
Management
Lack of Operational Transparency
Management failed to provide details on data storage and algorithms.
Inadequate Access Controls
Ministry of Security allowed unauthorized queries on biometric data.
Regulatory Environment
Absence of Functioning Controls
Good legislation existed but lacked active, independent enforcement.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
