This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
An analysis by the advocacy group Avaaz found that Facebook's AI-driven misinformation detection system failed to label 42% of posts containing debunked claims about COVID-19 and the 2020 U.S. election. The study revealed that bad actors could easily bypass the system by making minor modifications to images or text, such as changing fonts or backgrounds. These unlabeled posts reached an estimated 142 million views, highlighting significant gaps in the platform's automated moderation capabilities.
Avaaz, an international advocacy group, released a review of Facebook's misinformation identifying software showing that the labeling process failed to label 42% of false information posts, most surrounding COVID-19 and the 2020 USA Presidential Election.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The AI's robustness failure directly enabled the widespread dissemination of false and misleading information regarding COVID-19 and the U.S. election.
The AI system failed to perform reliably under minor variations in input data, such as font changes or image cropping, allowing debunked misinformation to bypass detection.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the failure of Facebook's deployed AI duplicate detection system to robustly identify and label modified misinformation posts.
Added categories peak at 4: Severe (Epistemic Harm, Direct)
The failure of Facebook's AI-driven moderation system to detect modified duplicates of debunked misinformation allowed false claims about COVID-19 and the 2020 U.S. election to reach an estimated 142 million views. This vulnerability highlights how simple adversarial perturbations can bypass automated defenses, presenting notable challenges to information security, societal stability, and democratic processes.
The AI system performs a specific, well-defined function of detecting and labeling duplicate misinformation posts based on direct inputs without maintaining an active goal-directed state.
Minimal or No Risk: The AI system is an automated content moderation filter, which is analogous to 'simple applications like spam filters' and poses minimal direct risk under the EU AI Act framework.
The AI system's primary goal is to moderate platform content by identifying and labeling duplicate instances of debunked misinformation.
The report clearly outlines the AI system's role, the specific failure mode (lack of robustness to minor image/text modifications), and provides quantitative data from a structured study by Avaaz.
The incident could be viewed purely as a human moderation policy failure rather than an AI failure, but the report explicitly highlights the AI's inability to detect modified duplicates as the core technical gap.
Technology
AI Sensitivity to Minor Tweaks
AI fails to match near-identical images with minor crops or background changes.
Text and Font Recognition Limits
Altering fonts or converting image text to status updates bypasses detection.
Inability to Track Mutated Memes
System cannot identify mutated versions of previously debunked viral memes.
Data Inputs
Incomplete Reference Database
Lack of diverse variations of debunked content in the match database.
Unlabeled Variation Data
Slightly altered posts lack the labels needed to train detection models.
Delayed Fact-Checking Inputs
Fact-checker data arrives after variations have already gone viral.
Human Factors
Orchestrated Adversarial Behavior
Bad actors actively edit content to deliberately exploit system vulnerabilities.
User Sharing Behaviors
Users rapidly share unlabeled misinformation, increasing viral reach.
Manual Review Bottlenecks
Human reviewers cannot keep pace with the volume of mutated variations.
Process and Methods
Inconsistent Labeling Workflows
Identical claims receive different warning labels or none at all.
Flawed Duplication Matching
The process for matching duplicate posts fails when minor edits are made.
Inadequate Repeat Offender Tracking
Failure to consistently flag pages that repeatedly post misinformation.
Management
Failure to Mitigate Known Gaps
Management failed to close loopholes despite knowing actors bypass systems.
Inadequate Policy Enforcement
Enforcement actions against repeat offender pages were inconsistent.
Regulatory Environment
Lack of External Standards
Absence of regulatory standards for social media misinformation detection.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
