This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
An investigation by Stanford University and the University of Massachusetts Amherst revealed that Instagram's recommendation algorithms were actively promoting accounts facilitating the sale of child sexual abuse material. Additionally, the study found that Twitter's automated scanning for known CSAM had ceased functioning following changes at the company. These failures in AI-driven content moderation and recommendation systems created significant risks for minors.
An investigation disclosed that Instagram's recommendation algorithms are promoting accounts that facilitate and sell child sexual abuse material (CSAM). The study, conducted by The Wall Street Journal and researchers at Stanford University and the University of Massachusetts Amherst, indicates that Instagram's algorithms not only allow for the discovery of such accounts through keyword searches but also actively recommend them to users within the network. The issue is especially concerning given Instagram's popularity among teenagers.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
Twitter's automated scanning for known CSAM broke and failed to perform reliably, representing a severe lack of capability and robustness in content moderation.
The Instagram recommendation algorithm actively promoted accounts facilitating child sexual abuse material, exposing users to highly toxic, illegal, and harmful content.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the post-deployment behavior of Instagram's recommendation algorithm and the failure of Twitter's automated scanning system, which produced unintended harmful outcomes.
Added categories peak at 4: Severe (Child Sexual Exploitation & Abuse (CSEA), Direct)
Failures in Instagram's autonomous recommendation algorithms and Twitter's broken automated scanning systems facilitated the promotion and discovery of child sexual abuse material networks. While representing a severe human rights violation and a significant regulatory challenge, the national security impact remains minor and manageable within standard law enforcement and legal frameworks.
The Instagram recommendation algorithm operates autonomously to suggest accounts to users in real-time based on their viewing patterns, without direct human intervention or explicit planning for each recommendation.
Unacceptable Risk: The report describes exploitative systems targeting vulnerabilities based on age (minors) and facilitating child sexual exploitation, which violates fundamental rights.
Instagram's system was designed for Content Recommendation, which actively promoted CSAM accounts, while Twitter's system was designed for Content Moderation (scanning for CSAM) and failed to function.
The report clearly details the algorithmic failures of both Instagram and Twitter regarding CSAM promotion and scanning, backed by a Stanford University study. The roles of the AI systems are explicitly stated, leaving little ambiguity about the nature of the failures.
None
Technology
Recommendation Algorithm Promotion
Algorithms promoted SG-CSAM accounts to users viewing network accounts.
Broken CSAM Scanning
Twitter's basic scanning for known CSAM broke after the takeover.
Widespread Hashtag Use
Widespread use of hashtags facilitated discovery of explicit accounts.
Data Inputs
Self-Generated CSAM Content
Accounts uploaded SG-CSAM pretending to be operated by minors.
Explicit Hashtags
Explicit hashtags were used to label and find illicit accounts.
Restricted API Data Access
Twitter cut off API access, blocking research into platform data.
Human Factors
Active Buyer and Seller Network
Malicious actors actively used platforms to buy and sell CSAM.
High Teen Platform Popularity
Instagram's popularity among teens made it a target ecosystem.
Process and Methods
Algorithmic Account Suggestion
Recommendations allowed discovery without requiring keyword searches.
Inadequate Content Review
Accounts offering CSAM remained active for months without removal.
Management
Delayed Safety Action
Meta set up an internal task force only after research was published.
Failure to Fulfill Pledges
Twitter failed to address CSAM despite executive pledges to do so.
Blocking External Researchers
Management cut off API access for researchers studying CSAM.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
