This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A Māori woman was wrongfully accused of trespassing at a New World supermarket in New Zealand after a facial recognition system incorrectly identified her as a known offender. The incident, which occurred on her birthday, caused significant emotional distress and public embarrassment. The supermarket acknowledged the error as a failure of human verification protocols and apologized, while experts highlighted concerns regarding racial bias and the accuracy of facial recognition technology for people of color.
A facial recognition system at New World Westend supermarket misidentified a Māori woman as a known offender during its trial. The woman was wrongfully accused of trespassing and experienced public embarrassment, raising concerns about racial bias and the technology's accuracy. The supermarket acknowledged its error and apologized.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
Supermarket staff over-relied on the AI's alert and failed to properly verify the match before confronting and accusing the customer.
The incident represents unequal performance of the facial recognition system across demographic groups, as experts highlighted that the technology is less accurate for people of color, specifically Maori and Pasifika.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was triggered by a false match generated by the deployed facial recognition AI system, leading to an unexpected and harmful outcome.
Added categories peak at 2: Minor (Psychological Harm, Direct)
A Maori woman was falsely accused of trespassing due to a false positive in a commercial facial recognition trial in New Zealand. The incident highlights ongoing concerns regarding demographic bias in biometric technology and privacy rights, but presents negligible direct threat to national security.
The facial recognition system acts as an Assistant because it flags potential matches to augment the staff's ability to identify trespassers, but explicitly requires human verification ('two trained team members must verify the matched image') before any action is taken.
High Risk: The report describes a facial recognition system used for identification and surveillance in public/semi-public spaces, which has significant implications for fundamental rights and public interest.
The system's primary function is to scan and match customers' faces against a database of known offenders.
The reports provide a clear, detailed account of the specific incident, including perspectives from the affected individual, the supermarket chain, and external experts. The role of the AI system (facial recognition) and the nature of the failure are explicitly documented.
The supermarket chain framed the incident primarily as a 'human error' in verification, whereas experts and the victim highlighted the systemic racial bias and technical limitations of the AI system itself.
Technology
Demographic Bias in FRT Software
FRT is less accurate for people of color and women of color.
Offshore Training of AI Models
Software not trained on NZ population, reducing Maori accuracy.
Data Inputs
Poor Database Image Quality
Comparison image was of low quality, leading to match error.
Inadequate Database Verification
Database contained incorrect or poorly labeled suspect profiles.
Human Factors
Staff Confirmation Bias
Staff assumed the system match was correct without critical review.
Aggressive Staff Confrontation
Staff got in customer's face and demanded she leave immediately.
Failure to Verify Identity
Staff ignored three forms of physical ID offered by the customer.
Process and Methods
Flawed Verification Process
Staff failed to properly verify the match before acting on alert.
Lack of Clear Escalation Steps
No process to de-escalate or double-check when customer contested.
Management
Premature Trial Deployment
Deploying unproven, biased FRT across 25 supermarkets.
Inadequate Staff Training
Staff training was insufficient to prevent critical human errors.
Regulatory Environment
Unproven Tech in Retail Spaces
FRT is not a proven tool to reduce retail crime in supermarkets.
Lack of Strict FRT Regulations
Absence of strong legal limits on deploying invasive FRT trials.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
