This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A report by Graphika identifies at least 10,000 AI chatbots, some utilizing APIs from major providers like OpenAI, Anthropic, and Google, that are being used to promote eating disorders, self-harm, and the sexualization of minors. These chatbots are hosted on various platforms and are often customized by users to bypass safety filters. Experts warn that these interactions can exploit vulnerable teenagers and children, reinforcing destructive real-world behaviors.
At least 10,000 AI chatbots have allegedly been created to promote harmful behaviors, including eating disorders, self-harm, and the sexualization of minors. These chatbots, some jailbroken or custom-built, leverage APIs from OpenAI, Anthropic, and Google and are hosted on platforms like Character.AI, Spicy Chat, Chub AI, CrushOn.AI, and JanitorAI.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
Vulnerable teenagers rely on these chatbots as anorexia coaches or self-harm buddies, substituting them for professional mental health support.
The incident involves AI chatbots generating toxic and harmful content, specifically promoting eating disorders, self-harm, and the sexualization of minors.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk is primarily driven by human users intentionally customizing, jailbreaking, and deploying chatbots to promote self-harm, eating disorders, and child exploitation.
Added categories peak at 3: Substantial (Psychological Harm, Direct)
An investigation revealed over 10,000 AI chatbots utilizing major APIs to promote self-harm, eating disorders, and the sexualization of minors. While presenting severe law enforcement and child safety concerns, the incident has minor direct implications for national security, manageable under standard regulatory and policing frameworks.
The chatbots function as interactive assistants, acting as 'anorexia coaches' or 'self-harm buddies' to provide companionship and advice to users.
Unacceptable Risk: The report describes AI systems being used to target and exploit the vulnerabilities of minors and individuals with mental health issues, promoting self-harm and eating disorders.
The AI systems are deployed as interactive conversational chatbots designed to roleplay specific personas.
The Graphika report provides clear, detailed evidence of the scale (10,000+ chatbots) and specific platforms involved. The types of harms (self-harm, eating disorders, sexualization of minors) are explicitly documented with expert commentary.
None
Technology
Bypassing Safety via Prompts
Simple prompts roleplaying future scenarios bypass safety measures.
Immersive 24/7 Chatbot Availability
Constant mobile access to chatbots makes harmful interactions highly immersive.
API Key and Model Sharing
Users trade API keys and customized models on forums to build harmful bots.
Data Inputs
NSFW and NSFL Dataset Usage
Models are optimized using datasets containing NSFL and NSFW content.
Human Factors
Vulnerability of Teenagers
Teens seeking validation are easily exploited by supportive chatbot personas.
Seeking Judgment-Free Spaces
Users seek non-critical environments to express harmful urges without stigma.
Lack of Critical Reasoning
Children struggle to distinguish between virtual bots and real people.
Process and Methods
Inadequate Platform Moderation
Startups have varied incentives and methods to moderate platform abuse.
Techniques for Jailbreaking
Communities actively share and develop methods to jailbreak safety filters.
Management
Resource Constraints in Startups
Small startups lack the resources and experience to handle platform abuse.
Regulatory Environment
Lack of Legal Consensus on Virtual CSAM
Belief that virtual CSAM is victimless reduces deterrence of its creation.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
