This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Families of two young girls, aged 8 and 9, filed a lawsuit against TikTok, alleging that the platform's recommendation algorithm intentionally pushed 'blackout challenge' videos to their children. The suit claims the algorithm's design prioritized engagement and profit over user safety, leading the children to attempt the dangerous choking game. TikTok has denied that the challenge was a platform trend, stating it predates the app, while researchers have separately documented the algorithm's tendency to serve harmful content to vulnerable accounts.
TikTok’s recommendation algorithm was alleged in a lawsuit to have intentionally and repeatedly pushed videos of the “blackout” challenge onto children’s feeds, incentivizing their participation which ultimately resulted in the death of two young girls.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The reports describe how the children became addicted to the platform and attempted dangerous physical challenges due to algorithmic engagement loops and overreliance.
The TikTok algorithm repeatedly exposed young children to highly toxic, unsafe, and lethal content, specifically self-harm, suicide-related videos, and dangerous physical challenges.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The harm resulted from the post-deployment operation of TikTok's recommendation algorithm, which unintentionally promoted lethal self-harm challenges to minors as a consequence of optimizing for user engagement.
Added categories peak at 4: Severe (Psychological Harm, Inferred)
An incident involving TikTok's recommendation algorithm autonomously serving lethal self-harm challenges to minors. While causing tragic individual fatalities and highlighting systemic platform safety issues, the national security impact remains minor and is managed through standard regulatory and legal processes.
The TikTok recommendation algorithm operates autonomously to curate the 'For You' feed in real-time. It uses learned patterns from user interactions (likes, watch time) to independently decide which videos to serve next without direct human intervention.
Unacceptable Risk: The report describes an AI system that intentionally and repeatedly pushed dangerous content to children, exploiting their vulnerabilities and leading to fatal outcomes, which aligns with 'exploitative systems targeting vulnerabilities based on age'.
The primary function of the TikTok algorithm is to recommend content to users on their 'For You' feed based on behavioral modeling of their preferences and interactions.
The reports provide clear, detailed accounts of the lawsuits, the specific victims, and the CCDH empirical study on TikTok's algorithm. The role of the recommendation algorithm in serving the harmful content is explicitly stated and documented.
The deaths could be viewed as a result of peer pressure or external internet trends rather than being solely driven by TikTok's algorithm.
Technology
Addictive Recommendation Algorithm
Algorithm designed to maximize engagement, pushing extreme content rapidly.
Algorithmic Promotion of Challenges
System pushed dangerous blackout challenge videos directly to children's feeds.
Rapid Content Amplification
Algorithm served self-harm videos to teens within minutes of account creation.
Data Inputs
User Metadata Exploitation
Usernames with terms like 'loseweight' triggered more harmful recommendations.
User Engagement Signals
Pausing or liking videos was ingested to serve more extreme content.
Demographic Targeting
Algorithm targeted young children based on age and profile demographics.
Human Factors
Childhood Cognitive Development
Young children could not comprehend the lethal risks of the challenges.
Desire for Online Fame
Children performed dangerous challenges hoping to become 'TikTok famous'.
Addictive User Behavior
Children became highly addicted to the continuous stream of recommended videos.
Process and Methods
Inadequate Age Verification
Children under the minimum age of 13 easily accessed and used the platform.
Flawed Content Moderation
Platform failed to detect and remove lethal challenge videos effectively.
Lack of Parental Warnings
System failed to warn parents about addictive features and dangerous trends.
Management
Prioritizing Profit Over Safety
Company prioritized user engagement and corporate profits over child safety.
Ignoring Known Safety Risks
Management knew the blackout challenge was spreading but failed to stop it.
Regulatory Environment
Weak Online Safety Laws
Lack of enforced regulations allowed platforms to host harmful content.
Lack of Platform Accountability
No legal frameworks held algorithms liable for promoting dangerous challenges.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
