This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Facebook's automated content moderation system incorrectly flagged and removed posts from users mentioning the Plymouth Hoe landmark, misidentifying the word 'Hoe' as a misogynistic slur. This resulted in users receiving warnings, content removals, and temporary bans from the platform. Facebook acknowledged the error and apologized for the impact on residents and visitors.
Facebook's automated system mistakenly labelled posts featuring the seafaring landmark Plymouth Hoe as misogynistic.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
The automated moderation system failed to perform robustly under varying linguistic contexts, misinterpreting a geographical landmark name as a slur due to a lack of contextual capability.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by an automated content moderation AI system acting post-deployment, which unintentionally flagged benign historical terms as offensive slurs.
An automated content moderation error on Facebook temporarily restricted UK users for mentioning the landmark 'Plymouth Hoe', misidentifying it as a slur. The incident represents a minor commercial platform glitch with negligible national security implications.
The Facebook automated moderation system acted autonomously to scan, flag, and remove user posts and issue bans without direct human intervention for each decision. The report notes 'Facebook are saying it's harassment and muting people and giving them a Facebook ban. It's not us doing that' indicating autonomous action.
Minimal or No Risk: The system is an automated content moderation/spam filter tool used on a social media platform, which poses low or negligible risk to users and society. According to the definitions, 'AI systems used in entertainment or simple applications like spam filters' fall under Risk Level 4.
The AI system's primary function was to moderate user-generated content on Facebook and flag potential harassment or offensive language.
The report clearly outlines the nature of the AI failure, the specific platform involved, the real-world impact on users, and the developer's response. There is no conflicting information or significant ambiguity regarding what transpired.
None
Technology
Overly Aggressive Filter
The automated moderation system flagged 'Hoe' without analyzing context.
Contextual Blindness
AI failed to recognize 'Plymouth Hoe' as a historical geographical location.
Data Inputs
Lack of Localized Training Data
Training dataset did not include UK geographical terms like 'Hoe'.
Unrefined Slur Dictionary
The dictionary listed 'hoe' as a slur without homonym disambiguation.
Human Factors
User Frustration and Confusion
Users had to bypass filters using spaces or spelling variations.
Over-reliance on Automation
System administrators trusted the automated flags without manual verification.
Process and Methods
Inadequate Post-Deployment Review
Lack of real-time monitoring to catch widespread false positives quickly.
Deficient Appeal Resolution
Muted users faced automated bans without rapid human appeal channels.
Management
Inadequate Risk Assessment
Management did not assess the risk of blocking common geographic terms.
Regulatory Environment
Pressure to Moderate Slurs
External pressure led to aggressive filtering of potential harassment terms.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
