This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Researchers from Global Witness and NYU's Cybersecurity for Democracy tested the automated moderation systems of Facebook, TikTok, and YouTube by submitting advertisements containing real-world death threats against US election workers. While TikTok and YouTube rejected the ads and suspended the accounts, Facebook's automated system approved 75% of the submitted ads. The researchers highlighted that this failure to detect overt violent language poses significant risks to the safety of election workers and the integrity of democratic processes.
Facebook's automated moderating system failed to flag and allowed ads containing explicit violent language against election workers to be published.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The system's failure to block the ads created a direct risk of exposing platform users to extreme violence and death threats.
The reports highlight a parallel failure where the same automated systems approved election disinformation ads.
The automated moderation system failed to detect overt death threats and violent ads, approving 15 out of 20 of them due to a lack of capability and robustness in its natural language processing.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk was caused by the automated moderation AI system failing to detect violent ads, which was an unexpected outcome of its deployment.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
A red-teaming investigation revealed that Facebook's automated ad moderation system approved 75% of test ads containing explicit death threats against US election workers. While the ads were deleted before publication to prevent actual harm, the failure highlights a minor but notable vulnerability in information security and election integrity safeguards.
The AI system is an automated moderation filter that performs the specific, well-defined function of reviewing and approving or rejecting submitted advertisements based on direct inputs without maintaining an ongoing context or planning.
High Risk: The report describes an AI system used in a high-stakes context (moderating election-related content) where failures have significant implications for public safety, fundamental rights, and democratic processes.
The AI system's primary goal is to review and moderate submitted advertisements to ensure they comply with platform policies and do not contain harmful content like death threats.
The reports provide clear, empirical results from a structured audit conducted by Global Witness and NYU C4D. The methodology, including the number of ads submitted (20) and approved (15), is explicitly detailed. There is no ambiguity about the AI system's failure to detect the violent content.
None
Technology
Flawed Automated Moderation
Facebook's automated system failed to flag explicit death threat text in ads.
Poor Image-Text OCR Integration
System failed to extract and analyze violent text overlaid on images.
Language Detection Variance
System performed inconsistently across English and Spanish language ads.
Data Inputs
Grammar and Profanity Sensitivity
System bypassed checks when profanity was removed and grammar was corrected.
Text-on-Image Format
Threat text was embedded in images, bypassing text-only classifiers.
Human Factors
Adversarial Editing of Ads
Researchers bypassed filters by removing profanity and correcting grammar.
Process and Methods
Inadequate Pre-Election Screening
Moderation processes failed to secure high-risk ads right before elections.
Lack of Keyword Triggering
Automated system lacked simple keyword flags for manual review of violent terms.
Post-Live Review Reliance
Facebook relied on post-live reviews instead of robust pre-publication checks.
Management
Under-resourced Content Moderation
Meta failed to adequately resource and fund global safety and integrity systems.
Overreliance on Automation
Management relied heavily on automated systems without enough human oversight.
Regulatory Environment
Lack of Binding Ad Standards
EU regulation is emerging but global binding standards remain weak or absent.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
No population impact data reported.
