This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A two-year study by the Institute for Strategic Dialogue found that Facebook's AI-supported content moderation systems failed to effectively identify and remove extremist content in Somali, Kiswahili, and Arabic. This failure allowed terrorist groups to openly share propaganda and calls for violence, which researchers warn could threaten democratic stability and public safety in East Africa, particularly during election periods. The report highlights a persistent gap in language parity for moderation, noting that the system also incorrectly flags non-terrorist content.
Facebook's system involving algorithmic content moderation for East African languages was reportedly failing to identify violating content on the platform such as mistakenly classifying non-terrorist content.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The effectiveness of the content moderation system was significantly lower for Swahili, Somali, and Arabic speakers compared to English speakers.
The AI system failed to perform reliably under non-English language conditions, leading to significant moderation blind spots.
The AI system failed to filter out extremist propaganda and calls to violence, thereby exposing users to toxic and harmful content.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The content moderation failures were caused by the AI system's inability to correctly identify extremist content after it was deployed.
A systemic failure in Meta's AI-driven content moderation allowed terrorist groups like al-Shabaab and ISIS to spread propaganda in Swahili, Somali, and Arabic. This failure compromised information security and threatened democratic stability and public safety in East Africa, particularly during election periods, highlighting critical gaps in regional language AI capabilities.
The content moderation AI operates independently to scan, classify, and filter user posts in real-time based on learned patterns without human intervention for each decision.
Minimal or No Risk: The AI system is used for content moderation, which is functionally similar to spam filtering and does not fall under the high-risk or unacceptable risk categories defined in the Act.
The primary goal of the AI system is to moderate social media content by identifying and removing policy-violating material, specifically extremist propaganda and hate speech.
The reports provide detailed findings from a structured two-year study by the Institute for Strategic Dialogue, with specific metrics on accounts, views, and languages. The AI's role in content moderation and its specific failure modes are clearly documented, allowing for high-confidence classifications.
The failure could be viewed primarily as an organizational resourcing issue rather than an AI capability failure, though the AI's technical limitations in language processing are central to the report.
Technology
AI Language Moderation Gaps
AI failed to recognize extremist content in Swahili, Somali, and Arabic.
Incorrect Content Classification
AI mistakenly classified non-terrorist content while missing actual threats.
Failure to Detect Known Branding
Algorithms failed to flag official Al-Shabaab branding on uploaded videos.
Data Inputs
Lack of Regional Dialect Data
Insufficient training data for Arabic dialects and East African languages.
Evasion via Account Reconstitution
Terrorists bypassed detection by recreating banned pages under new names.
Human Factors
Inadequate Moderator Training
Moderators and outsourced entities lacked training on regional terror content.
Lack of Native Language Speakers
Insufficient human moderators fluent in Swahili, Somali, and local dialects.
Process and Methods
Weak Account Verification Processes
Lax verification allowed banned users to easily set up duplicate profiles.
Slow Incident Response Times
Flagged extremist videos remained online for months before being taken down.
Management
Prioritizing Growth Over Safety
Platform prioritized expansion over investing in localized safety mechanisms.
Underinvestment in At-Risk Regions
Inadequate resources allocated to content moderation in non-English languages.
Regulatory Environment
Lack of Regional Oversight
Absence of strong local regulatory pressure on social platforms in East Africa.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
