This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
This report details two separate AI-related incidents involving minors. First, Amazon's Alexa voice assistant suggested a dangerous 'penny challenge' to a ten-year-old girl. Second, research by the Center for Countering Digital Hate found that TikTok's recommendation algorithm actively promotes self-harm, suicide, and eating disorder content to accounts registered as 13-year-olds, with vulnerable accounts receiving such content at significantly higher rates.
Amazon’s voice assistant Alexa suggested “the penny challenge,” which involves dangerously touching a coin to the prongs of a half-exposed plug, when a ten-year-old girl asked for a challenge to do.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
Amazon's Alexa failed to perform robustly by pulling a highly dangerous physical challenge from the web and recommending it to a child.
The AI systems exposed minor users to highly toxic and dangerous content, including a life-threatening physical challenge and videos promoting self-harm, suicide, and eating disorders.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incidents were caused by deployed AI systems (Alexa and TikTok's recommendation algorithm) generating unexpected and harmful outputs post-deployment.
Added categories peak at 3: Substantial (Psychological Harm, Inferred)
The incidents involve consumer AI systems exposing minors to physical and psychological risks, specifically Amazon Alexa suggesting a dangerous physical challenge and TikTok's algorithm systematically promoting self-harm content to vulnerable teens. While primarily public safety issues, the scale of TikTok's algorithmic impact on Western youth represents a substantial societal stability concern.
The report describes Amazon's Alexa as a 'voice assistant' designed to provide information and suggestions upon user request. TikTok's recommendation algorithm also acts to suggest content based on user preferences, requiring user interaction to function.
Unacceptable Risk. The report describes TikTok's recommendation algorithm as an exploitative system that targets the vulnerabilities of teenagers based on their age and mental state, bombarding them with harmful self-harm and suicide content.
Alexa is explicitly described as a voice assistant that answers user queries, while TikTok's algorithm functions to recommend content to users based on their interaction history.
The reports provide clear, detailed accounts of both the Alexa incident and the CCDH study on TikTok. The facts are straightforward, and the roles of the AI systems are explicitly defined.
The TikTok algorithm's behavior could be interpreted as a design optimization for engagement rather than intentional exploitation, though the outcome remains highly harmful.
Technology
Unfiltered Web Scraping
Assistant pulled dangerous instructions directly from the web.
Engagement-Driven Algorithms
Recommendation engine prioritized watch-time over user safety.
Lack of Real-time Safety Guardrails
System failed to block hazardous physical challenge recommendations.
Data Inputs
Raw Search Index Data
System used unvetted web source data for answering user queries.
User Behavioral Signals
Brief pauses and likes on videos triggered harmful recommendations.
Human Factors
User Curiosity and Vulnerability
Minor users sought out challenges and self-harm content online.
Lack of Parental Supervision
Children interacted with AI devices and apps unsupervised.
Process and Methods
Inadequate Content Filtering
Platforms failed to proactively block known dangerous challenges.
Deficient Safety Testing
Voice assistant was not properly tested against unsafe web content.
Management
Reactive Safety Mitigation
Action was taken only after public exposure of the safety incidents.
Prioritizing Platform Growth
Focus on user engagement led to insufficient safety oversight.
Regulatory Environment
Weak Online Safety Regulations
Absence of strict legal mandates to prevent exposure of minors to harm.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
