This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Facebook's automated content moderation system repeatedly flagged gardening-related posts as harassment or bullying due to the word 'hoe' and discussions about killing garden pests. This resulted in incorrect enforcement actions against a gardening group, causing concern among members and administrators. Facebook eventually corrected the mistaken enforcements and implemented a manual review step for the group.
Facebook's automated system flagged gardening groups' use of "hoe" and violent language against bugs as a violation by mistake.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
The AI system failed to perform reliably under normal conditions due to a lack of capability to understand linguistic context, leading to false-positive flags on benign words like 'hoe' and pest-control discussions.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by Facebook's deployed automated content moderation algorithm incorrectly flagging benign gardening terms and discussions as violations of community standards.
An automated content moderation failure on Facebook incorrectly flagged benign gardening terms as policy violations. This incident represents a minor, non-malicious AI safety failure with negligible implications for national security.
The report describes Facebook's automated moderation system operating independently to flag posts, delete comments, and disable commenting in real-time based on learned patterns, without requiring direct human execution for each action.
Minimal or No Risk: The AI system is used for automated content moderation, which is a low-risk application similar to spam filters, posing negligible risk to safety or fundamental rights.
The AI system's primary goal is to moderate content on Facebook by flagging and removing posts that violate community standards, specifically targeting hate speech, harassment, and violence.
The report clearly outlines the events, the AI system's role in flagging benign words, the impact on the gardening group, and Facebook's response. There is little ambiguity about what transpired.
None
Technology
Context-Blind NLP Algorithms
AI lacks understanding of linguistic nuance and context like gardening terms.
Overly Zealous Automation
Automated systems flag benign words like hoe without human verification.
Data Inputs
Homophone Confusions
Algorithms confuse the tool hoe with a phonetically identical slur.
Human Factors
User Frustration with Appeals
Group admins find navigating automated appeal surveys futile and useless.
Process and Methods
Flawed Automated Flagging Process
System flags posts and threatens group deletion without human review.
Inadequate Support Channels
Lack of accessible human-in-the-loop support for system errors.
Management
Neglect of Customer Support
Facebook failed to build adequate support and policy enforcement transparency.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
