This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Research by the Center for Countering Digital Hate (CCDH) indicates that TikTok's recommendation algorithm actively promotes pro-eating disorder, self-harm, and suicide-related content to users, including minors, shortly after they show interest in such topics. The platform's 'For You' page design creates a feedback loop that can trap vulnerable users in cycles of triggering content. While TikTok claims to consult with health experts and remove policy-violating content, the study suggests the algorithm's design facilitates the rapid spread of harmful material.
TikTok’s young users were allegedly exposed to community-guideline-violating pro-eating disorder content on their algorithmically curated “For You” page that serves videos from any user on its platform.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The algorithm's feedback loop creates personalized filter bubbles that trap vulnerable users in cycles of triggering content, degrading their information ecosystem.
The algorithm lacks the robustness to prevent users from bypassing content filters using simple spelling variations, leading to system failure in restricting harmful content.
The TikTok recommendation algorithm actively promotes and exposes young users to toxic and harmful content, including self-harm, eating disorders, and suicide-related material.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The recommendation algorithm's post-deployment optimization for engagement unintentionally led to the rapid promotion of self-harm and eating disorder content to vulnerable teenagers.
Added categories peak at 4: Severe (Psychological Harm, Inferred)
TikTok's recommendation algorithm was found to rapidly serve self-harm and eating disorder content to minors through automated feedback loops. While representing a severe global public health and societal safety challenge that has spurred regulatory action, the direct national security threat remains minor and is primarily a matter of societal resilience and regulatory policy.
The recommendation algorithm operates independently to curate the For You Page in real-time. It responds to user inputs like watch time, likes, and hashtags using learned patterns to decide which videos to serve next without human intervention.
Unacceptable Risk: The system acts as an exploitative system targeting vulnerabilities based on age (teenagers) and mental health conditions, using behavioral influence to promote self-harm and eating disorders.
The primary function of the TikTok algorithm is to recommend personalized video content to users on their For You Page based on their interests and behavior.
The reports provide detailed empirical evidence from a structured study by the CCDH, alongside expert commentary from psychologists, tech ethicists, and users. The role of the recommendation algorithm is explicitly defined and analyzed.
TikTok argues that the CCDH study does not reflect real-world user habits, suggesting the simulated accounts represent an extreme edge case rather than typical user experiences.
Technology
Optimizing for Engagement
Algorithm prioritizes watch time and likes over user safety.
Inability to Block Keywords
Platform lacks features for users to proactively block specific keywords.
Loophole-Prone Content Filtering
Moderation algorithms fail to detect modified spellings like thinsp0.
Data Inputs
User Engagement Indicators
Algorithm uses brief pauses and likes as signals to recommend content.
Bypassed Filter Terms
Users use terms like aliven't or suic1de to bypass data filters.
Triggering Recovery Videos
Videos showing body checks and weights act as harmful inputs.
Human Factors
Vulnerable User Behavior
Users with eating disorders seek out triggering content to compare.
Ineffective Trigger Warnings
Users ignore trigger warnings, failing to prevent exposure.
Well-Meaning Backlash
Users calling out bad content inadvertently increase its engagement.
Process and Methods
Lacking User Customization
Processes do not allow users to easily filter or customize feeds.
Ineffective Moderation Processes
Systems fail to quickly remove or flag harmful eating disorder content.
Lack of Local Partnerships
Difficulties partnering with local groups due to organizational gaps.
Management
Growth Over Safety Focus
Prioritizing never-ending scroll and engagement over user mental health.
Remote Management Structure
Lack of local offices in regions like Australia limits support.
Regulatory Environment
Lagging Regulatory Frameworks
Online Safety Bill is proposed but not yet active or fully enforced.
Unregulated Social Media
Lack of strict regulatory penalties for hosting age-inappropriate content.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
