This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A facial recognition pilot project in Bahia, Brazil, implemented by the government using technology from Iecisa and Huawei, has been linked to disproportionate targeting of Black citizens. The system, which suffers from high error rates and racial bias, has resulted in wrongful police stops and psychological harm to innocent individuals. Despite these issues, the government has expanded the surveillance network, raising significant concerns regarding algorithmic bias and civil rights.
Facial recognition deployed in a pilot project by the local government of Bahia despite having minimal hit rate reportedly targeted Black and poor people disproportionately.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
Police officers over-relied on the AI's automated alert, tracking an innocent citizen for 22 kilometers and conducting an invasive stop based solely on the system's suggestion.
The system suffers from a severe lack of robustness and accuracy, as evidenced by the report that only 3.6 percent of generated alerts led to actual arrest warrants.
The facial recognition system exhibits unequal performance across demographic groups, specifically suffering from higher error rates and false matches when processing Black faces compared to white faces.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The wrongful stop and tracking of PERSON_001 was caused by an erroneous match generated by the deployed facial recognition AI system, which was an unintended outcome of its operation.
Added categories peak at 2: Minor (Psychological Harm, Direct)
The deployment of a biased facial recognition system in Bahia, Brazil, highlights significant domestic societal stability and human rights concerns. The system disproportionately targets Black citizens due to high false-positive rates, leading to wrongful police stops and civil rights violations. While the threat to physical infrastructure is negligible, the state-wide expansion of foreign-sourced surveillance technology poses long-term concerns for societal trust and governance.
The AI system acts as an assistant to law enforcement by scanning public feeds and flagging potential matches, but it requires human review ('analise humana') and manual police intervention to act on those suggestions.
High Risk: The system is a law enforcement application utilizing facial recognition in public spaces, which is explicitly classified as High Risk under the EU AI Act.
The system's primary goal is to perform real-time facial recognition on public transit passengers to identify individuals matching a database of wanted suspects.
The report provides a detailed first-hand account of a specific victim (PERSON_001), statistical data on system error rates (3.6 percent accuracy), and expert analysis on algorithmic bias. The roles of the AI developers (Iecisa, Huawei) and the government are clearly defined, leaving little ambiguity about the nature of the failure.
None. The incident is clearly an AI-driven false positive resulting in a wrongful police stop.
While technical details of the proprietary algorithms would be useful, they are not necessary to confidently classify the incident and its harms.
Technology
Algorithmic Racial Bias
AI algorithms have higher error rates and false matches for Black individuals.
Inaccurate Facial Matching
System triggered a false alert at 90 percent similarity for an innocent person.
Inadequate Algorithm Training
Algorithm not trained on local demographics to prevent high false matches.
Data Inputs
Flawed Arrest Databases
Warrant databases contain outdated, incorrect, or racially biased records.
Unregulated Social Media Use
Police use public social media photos to feed the comparison database.
Biased Training Datasets
Datasets lack diverse representation, leading to racial misidentification.
Human Factors
Confirmation Bias in Review
Human operators validated the flawed 90 percent match instead of catching error.
Police Reliance on AI Alerts
Officers trusted the automated alert, leading to tracking and stop of citizen.
Process and Methods
Unrecorded False Positives
False identifications resolved on site are not recorded, hiding system errors.
Inadequate Verification Protocol
Lack of a rigorous method to verify system alerts before initiating police stop.
Management
Prioritizing Tech over Safety
State promoted facial recognition as a showcase despite low accuracy rates.
Lack of System Accountability
Management expanded the system without resolving high rate of false alerts.
Regulatory Environment
Lack of Surveillance Regulation
No clear legal framework limits facial recognition use in public spaces.
Absence of Data Privacy Controls
No strict controls on integrating private cameras or social media data.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
