This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Facebook's automated content moderation AI incorrectly flagged an advertisement for Walla Walla onion seeds from The Seed Company by E.W. Gaze as 'overtly sexual.' The system misinterpreted the round shape of the onions in the advertisement photo as suggestive. The advertisement was later reinstated after the company requested a human review, and Facebook acknowledged the error.
Facebook’s content moderation algorithm misidentified and removed a Canadian business’s advertisement containing a photo of onions as products of overtly sexual content, which was later reinstated after review.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
The AI system failed to perform reliably under normal conditions, misidentifying a benign image of onions as sexually suggestive content due to a lack of robustness in its computer vision capabilities.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by an unexpected false positive decision made by Facebook's automated content moderation AI after it was deployed.
A Facebook content moderation AI temporarily blocked an advertisement for onion seeds after misinterpreting the round shape of the onions as sexually suggestive. The incident was a minor technical error with zero national security implications, resolved quickly through human review.
The AI system operated independently to scan, flag, and remove the advertisement based on its automated nudity detection algorithms without requiring real-time human intervention or oversight prior to the action.
Minimal or No Risk: The AI system is an automated content moderation filter used to keep nudity off social media apps, which poses minimal risk to users and is analogous to simple applications like spam filters.
The AI system's primary goal was to automatically moderate content on Facebook's platform to keep nudity and sexually suggestive material off its apps.
The reports are highly consistent, clear, and explicitly state that an automated algorithm incorrectly flagged an onion advertisement as sexual. The details of the system's action, the company's reaction, and Facebook's response and apology are well-documented across multiple sources.
None
Technology
Errant Moderation Algorithm
AI failed to distinguish round onions from human nudity.
Over-zealous Computer Vision
Automated tech flagged non-sexual round shapes as suggestive.
Data Inputs
Ambiguous Visual Features
The shape of Walla Walla onions resembled human anatomy to the AI.
Human Factors
Lack of Immediate Human Review
No human verified the automated ban before it was enacted.
Process and Methods
Automated First-Pass Moderation
System automatically takes down flagged content without human check.
Delayed Appeal Process
No immediate reply or quick path to human review after flagging.
Management
Reliance on Automated Systems
High volume of moderation (35.7M removals) handled mostly by AI.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
