This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
TikTok's recommendation algorithm has been found to distribute content promoting eating disorders to its users, including those who have not actively sought out such material. Despite the platform's efforts to ban specific hashtags, users have successfully bypassed these restrictions using misspellings. The algorithm's engagement-based design creates a cycle that can trap vulnerable users in exposure to harmful content, raising significant concerns regarding the platform's safety procedures.
Videos promoting eating disorders evaded TikTok's automated violation detection system without difficulty via common misspellings of search terms, bypassing its ban of violating hashtags such as "proana" and "anorexia".
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The platform's search filters and content moderation systems lacked the robustness to prevent users from bypassing hashtag bans using simple misspellings.
The TikTok recommendation algorithm exposed users to harmful and unsafe pro-eating disorder content on their For You Pages, which constitutes exposure to toxic content.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The recommendation algorithm unintentionally distributed harmful eating disorder content to users post-deployment as an unexpected outcome of optimizing for engagement.
Added categories peak at 3: Substantial (Psychological Harm, Inferred)
TikTok's recommendation algorithm unintentionally distributed harmful eating disorder content to vulnerable young users due to engagement-based optimization. While raising significant public safety and corporate responsibility concerns, the incident has negligible direct national security implications.
The recommendation algorithm operates independently to select and display videos based on learned engagement patterns and real-time user interactions, without requiring direct human intervention.
Unacceptable Risk: The report describes an algorithm that creates a 'vicious cycle' of behavioral influence, trapping vulnerable young users by continuously showing them harmful eating disorder content, which aligns with systems causing behavioral influence leading to harm.
The primary function of the TikTok algorithm described is to recommend and curate video content for users' For You Pages based on engagement metrics.
The reports provide clear details about how the TikTok recommendation algorithm functions, the specific hashtags involved, and the platform's response. The role of the AI is explicitly defined as the primary driver of content distribution.
The incident could be viewed purely as a content moderation failure by human staff rather than an algorithmic failure, though the algorithm's active recommendation role makes the AI-centric interpretation highly compelling.
Technology
Recommendation Loop
FYP algorithm traps users in cycles of harmful eating disorder content.
Bypass via Search Terms
Users bypass hashtag bans using plain text search terms.
Circumvention via Misspellings
Users bypass keyword blocks by using common misspellings of banned words.
Data Inputs
Harmful Content Uploads
Pro-anorexia and pro-bulimia content uploaded directly by users.
Unlabeled User Metadata
Lack of precise metadata labels on harmful videos hinders filtering.
Human Factors
Vulnerable Target Audience
Large share of young users (16-24) susceptible to harmful body trends.
Active Search Circumvention
Users actively search for harmful terms to bypass content restrictions.
Process and Methods
Incomplete Filtering Rules
Restricting hashtags but leaving plain text search completely open.
Lack of Proactive Detection
Absence of automated tools to recognize and downrank self-harm content.
Management
Reactive Moderation Strategy
Banning accounts only after external investigations highlight issues.
Regulatory Environment
Lack of Algorithmic Standards
No external regulations governing algorithmic content recommendation safety.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
