This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Instagram's automated content moderation AI failed to detect and remove racist emoji-based abuse targeted at Black English football players after the Euro 2020 final, mistakenly marking the toxic comments as benign.
Facebook's, Instagram's, and Twitter's automated content moderation failed to proactively remove racist remarks and posts directing at Black football players after finals loss, allegedly largely relying on user reports of harassment.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The AI lacked the capability and contextual robustness to correctly identify emojis being used as racist markers.
The AI system's failure to detect and filter racist emojis directly exposed the targeted players and the public to toxic hate speech and harassment.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the failure of Instagram's deployed content moderation AI, which unintentionally misclassified racist emoji abuse as benign comments.
Added categories peak at 2: Minor (Psychological Harm, Direct)
Following the Euro 2020 final, Instagram's automated content moderation AI failed to detect and block racist emoji-based abuse targeting Black English football players. While the incident caused significant public distress, political debate, and highlighted moderation vulnerabilities, its direct national security impact remains minor and limited to societal and regulatory discussions.
The AI system acts as an Assistant by helping human reviewers prioritize reports and taking automated moderation actions, augmenting the human content moderation team.
Minimal or No Risk: The AI system is a content moderation filter, which is functionally similar to a spam filter and poses low or negligible risk to society under the EU AI Act definitions.
The AI system's primary function was to detect hate speech and moderate toxic content on the social media platform.
The reports provide clear, consistent details about the failure of Instagram's AI to detect racist emojis, the specific individuals targeted, and the response from Facebook executives and public figures. There is little ambiguity about the nature of the failure or the system involved.
None
Technology
Flawed Contextual AI
AI failed to understand the racist context of specific emoji combinations.
Mistaken Benign Classifications
AI mistakenly classified offensive emoji comments as benign.
Data Inputs
Lack of Emoji Abuse Data
System lacked training data on emojis used as racial markers.
Human Factors
Exploitative User Behavior
Trolls used emojis to evade standard text-based moderation filters.
Onus Placed on Victims
Victims and fans had to manually report abuse to trigger reviews.
Process and Methods
Reactive Moderation Model
The platform relied on reports instead of proactive emoji filtering.
Delayed Review Prioritization
AI misclassification delayed human review of critical emoji reports.
Management
Acceptance of Scale Gaps
Accepting a low error percentage resulted in thousands of missed abuses.
Inadequate Proactive Controls
Management failed to implement proactive filters for known emoji threats.
Regulatory Environment
Lack of Strict Mandates
No regulatory penalties existed to enforce immediate removal of abuse.
