This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Internal Facebook documents revealed that the company's AI-powered automated moderation tools were significantly less effective than claimed, failing to remove the vast majority of hate speech and violent content. The tools frequently misclassified severe content, such as mass shooting livestreams, and the company prioritized reducing false positives over increasing detection rates. This failure resulted in widespread exposure to harmful content, particularly in non-English speaking regions.
Facebook's automated moderation tools were shown by internal documents performing incomparably to human moderators, and accounting for only a small fraction of hate speech, violence, and incitement content removal.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The AI system performed unequally across linguistic groups, with a near-total failure to detect hate speech in Afghanistan (0.23%) compared to English-speaking regions.
The AI system demonstrated a severe lack of capability and robustness, misclassifying mass shootings as paintball and failing to generalize across different languages.
The primary failure of the AI moderation system resulted in widespread exposure of users to toxic content, including hate speech, violence, incitement, and mass shooting videos.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk of hate speech and violence slipping through was caused by the failure of the deployed AI moderation tools to perform robustly, which was an unintentional outcome of the system's technical limitations.
Added categories peak at 4: Severe (Psychological Harm, Inferred)
Internal leaks revealed that Facebook's AI moderation tools failed to remove 95-97% of hate speech and 99.4% of violence/incitement, with even worse performance in non-English regions like Afghanistan. While not a direct kinetic or state-sponsored threat, this systemic failure represents a substantial threat to global societal stability and human rights by permitting the unchecked spread of toxic content.
The AI system functions as a content classifier that performs the specific, well-defined task of identifying and filtering violating content based on direct inputs (posts and videos) without maintaining an ongoing state or planning.
Minimal or No Risk: The automated moderation tools function similarly to spam filters, which are classified as posing minimal or no risk under the EU AI Act.
The AI system's primary purpose is to proactively detect and remove content that violates Facebook's community standards, specifically hate speech and violent incitement.
The report is highly detailed, drawing from leaked internal Facebook documents and whistleblower testimony, providing specific statistics on the AI's failure rates (e.g., removing only 3-5% of hate speech views) and organizational context.
One could interpret the incident primarily as a corporate governance and public relations failure rather than a purely technical AI failure, given that management intentionally made user reporting harder to manipulate metrics.
Technology
Inability to Understand Context
AI confused cockfights with car crashes and mass shootings with paintball.
Low Recall of Integrity Harms
AI tools only caught 3-5 percent of views of hate speech on the platform.
Missing Local Language Support
Lack of dictionaries in languages like Dari and Pashto limited detection.
Data Inputs
Lack of Local Slur Dictionaries
No dictionaries for local Afghan languages hindered training data.
Flawed Evaluation Datasets
Estimations relied on small samples of posts to guess platform-wide harms.
Human Factors
User Reporting Barriers
Making it harder for users to report content falsely boosted AI performance.
User Safety Preferences Ignored
Users favored aggressive enforcement, but leadership ignored this preference.
Process and Methods
Ignoring User Reports
Algorithms were tweaked to ignore user reports to lower human review costs.
Misleading Performance Metrics
Using proactive detection rate to obscure low platform-wide recall.
Optimizing to Avoid False Positives
Training models to avoid false positives allowed hate speech to slip through.
Management
Prioritizing Cost Over Safety
Targeted a 15 percent reduction in human content moderation review costs.
Misleading Public Communication
Management refused to disclose metric calculations to external groups.
Focus on Avoiding Take-downs
Leadership forced engineers to prioritize avoiding false positives over safety.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
