This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Character.AI, a platform allowing users to create custom chatbots, was used to generate personas of George Floyd, a murder victim. These chatbots engaged in thousands of conversations, making false and controversial claims about his death and current status, such as being in witness protection. The company subsequently flagged the chatbots for removal following reports, citing their safety policies.
Two chatbots emulating George Floyd were created on Character.ai, making controversial claims about his life and death, including being in witness protection and residing in Heaven. Character.ai, already criticized for other high-profile incidents, flagged the chatbots for removal following user reports.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The chatbots generated false claims about George Floyd's death being faked and him being in witness protection, spreading misinformation to users.
The creation of chatbots mimicking a murder victim with offensive taglines and conspiracy theories represents toxic content that violates community norms and platform guidelines.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was initiated by human users who intentionally created offensive and unauthorized chatbots mimicking a deceased murder victim.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
Users created offensive AI chatbots of George Floyd on Character.AI that spread false narratives. While the incident touches on sensitive societal themes, it constitutes a standard platform trust and safety violation rather than a coordinated national security threat, resulting in minor societal impact and negligible threat to national infrastructure or sovereignty.
The AI system operates as a conversational chatbot designed to emulate a specific persona and engage in discussions with users based on user inputs and prompts.
Risk Level 3. Limited Risk: The system is a chatbot platform (Character.AI) where users interact with AI-generated personas, which falls under the category of chatbots and AI-generated content.
The primary function of the Character.AI platform is to allow users to create and interact with customizable chatbots. The chatbots in this incident also utilized synthetic voice generation to mimic a specific voice.
The report clearly details the existence of the chatbots, their specific outputs, the number of interactions, and the platform's response. There is little ambiguity regarding the facts of the incident.
None
Technology
Generative Hallucinations
Chatbots claimed faked deaths and witness protection, spreading misinformation.
Inadequate Proactive Filters
System failed to block creation of insensitive personas before public access.
Bypassed Safety Guidelines
Though some answers were blocked, offensive chatbots remained active.
Data Inputs
Unverified User Persona Inputs
Platform accepted user-defined tags and names of real murder victims.
Unfiltered Training Mimicry
AI voice and text mimicked deceased individuals based on user-provided data.
Human Factors
Malicious User Intent
Users deliberately created offensive personas of deceased tragedy victims.
User Engagement with Harmful Bots
Over 13,000 interactions occurred, showing high engagement with harmful bots.
Process and Methods
Reactive Moderation Model
Platform relied heavily on user flagging and reports to remove harmful bots.
Insufficient Proactive Detection
Custom blocklists failed to catch variations of sensitive names and taglines.
Management
Prioritizing Growth over Safety
Allowing open user creation without strict pre-release vetting processes.
Inadequate Risk Assessment
Failure to anticipate and prevent the weaponization of deceased personas.
Regulatory Environment
Lack of Platform Liability Laws
Absence of strict laws holding platforms liable for user-generated AI bots.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
