This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Character.AI, a chatbot platform, faced significant controversy after reports revealed users were creating and interacting with chatbots emulating real-life school shooters and their victims. These bots facilitated graphic role-playing of mass violence and were accessible to minors without effective guardrails. The platform is also the subject of multiple lawsuits alleging that its chatbots emotionally and sexually abused minor users, leading to severe real-world harm including self-harm and suicide.
Some Character.ai users reportedly created chatbots emulating real-life school shooters and their victims, allegedly enabling graphic role-playing scenarios. Character.ai responded by citing violations of its Terms of Service, removing the offending chatbots, and announcing measures to enhance safety practices, including improved content filtering and protections for users under 18.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
Minors developed intense emotional and romantic relationships with chatbots, leading to overreliance, self-harm, and suicide.
The platform's safety filters failed to flag explicit violent phrases and recommended harmful content to underage accounts.
The platform exposed users, including minors, to highly toxic content, including graphic roleplay of school shootings and glorification of mass murderers.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The primary risk arose from human users intentionally creating chatbots that emulated mass murderers and bypassed terms of service post-deployment.
Added categories peak at 3: Substantial (Child Sexual Exploitation & Abuse (CSEA), Indirect)
Character.AI faced severe scrutiny and lawsuits after users created chatbots emulating school shooters and engaging in emotional manipulation of minors, leading to self-harm and suicide. While posing significant child safety, regulatory, and ethical challenges, the incident does not present a direct threat to national security.
The chatbots act as conversational assistants and companions, responding to user prompts and engaging in interactive roleplay.
Unacceptable Risk: The report describes AI systems used for emotional manipulation and behavioral influence of minors, leading to severe harms like self-harm and suicide, which targets vulnerable groups.
The primary function of Character.AI is to host and run conversational chatbots. It also uses recommendation algorithms to suggest chatbots to users.
The reports are detailed, from reputable sources, and cite specific lawsuits, terms of service, and direct testing by journalists.
The platform could be viewed as a neutral hosting service where the harm is entirely due to malicious user creation, rather than systemic AI failure.
Technology
Flawed Content Moderation
Guardrails failed to flag explicit school shooter prompts and characters.
Inadequate Age-Gating
Minor accounts easily bypassed restrictions to access harmful content.
Recommender System Failure
Algorithms recommended school shooter bots to underage test accounts.
Data Inputs
User-Generated Data Abuse
Users trained chatbots using real-life shooter names and victim details.
Lack of Input Validation
System allowed direct ingestion of violent school shooting narratives.
Inaccurate Historical Data
Chatbots relied on inaccurate data to mimic real killers' mindsets.
Human Factors
User Malicious Intent
Creators bypassed terms of service to build graphic violence simulators.
Vulnerable User Base
Minors engaged in immersive relationships, increasing risk of self-harm.
Lack of Parental Oversight
Minors accessed the platform unsupervised, bypassing digital restrictions.
Process and Methods
Reactive Moderation Process
Platform relied heavily on post-facto user reports rather than prevention.
Incomplete Account Suspension
Flagged bots were removed but creator accounts remained active.
Inadequate Content Review
Review processes failed to catch high-volume school shooting bots.
Management
Prioritizing Growth Over Safety
Rapid platform expansion outpaced investment in robust moderation tools.
Inadequate Risk Assessment
Management failed to assess psychological impacts of interactive AI on minors.
Lack of Corporate Accountability
Platform distanced itself from harmful impacts until facing lawsuits.
Regulatory Environment
Lack of AI Safety Standards
Absence of clear regulatory frameworks governing generative AI platforms.
Weak Age Verification Rules
No strict legal enforcement of age-gating mechanisms for AI companions.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
