This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Users of the Replika AI chatbot app have been documented engaging in abusive, violent, and sexually harassing interactions with their AI companions. These users frequently share these interactions on social media platforms like Reddit. While the AI itself cannot experience suffering, experts express concern that this behavior normalizes domestic abuse, reinforces toxic masculinity, and may negatively impact the mental health of users who are psychologically reliant on the chatbots.
Replika's AI-powered "digital companions" was allegedly abused by their users, who posted on Reddit abusive behaviors and interactions such as using slurs, roleplaying violent acts, and stimulating sexual abuse.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
Also classified as
The female-coding of the chatbots and their passive responses to abuse reinforce harmful gender stereotypes and unequal power dynamics.
Users generate and are exposed to highly toxic, sexually explicit, and verbally abusive content during their interactions and on associated forums.
The incident centers on users anthropomorphizing and forming deep romantic or emotional attachments to AI chatbots, leading to overreliance, psychological dependence, and unsafe dynamics like simulated domestic abuse.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The primary risk and abusive behaviors are driven by the decisions and actions of human users interacting with the AI.
Added categories peak at 3: Substantial (Psychological Harm, Inferred)
The incident involving user abuse of the Replika chatbot highlights evolving risks in human-AI interaction, psychological dependency, and the normalization of toxic behaviors. However, it presents negligible direct threats to physical security, intelligence, sovereignty, or economic security, resulting in a minor overall national security impact.
Replika functions as an AI companion, friend, or mentor that responds to user inputs to simulate human-like conversation, requiring user interaction to drive the dialogue.
Risk Level 3: Limited Risk. The Replika system is a chatbot, which falls under the Limited Risk category requiring transparency obligations so users are aware they are interacting with an AI.
The primary function of Replika is to act as a conversational chatbot that simulates human relationships and interactions.
The reports provide consistent, detailed accounts from multiple reputable media outlets (Futurism, Fortune, Jezebel) regarding the behavior of Replika users, the design of the app, and the ethical concerns raised by experts. The facts regarding user behavior, app statistics, and expert opinions are well-documented and aligned across sources.
Some might interpret the user behavior as harmless roleplay or catharsis rather than a risk of normalizing real-world abuse.
Technology
Submissive AI Response Defaults
Passive or pleading responses from the bot encourage continued user abuse.
Female-Coded Default Avatars
Gendered design reinforces misogynistic stereotypes and power dynamics.
Realistic Conversational Mimicry
Advanced language generation facilitates deep emotional projection and abuse.
Data Inputs
Lack of Input Moderation Filters
Replika lacked systems to detect and block abusive user text inputs.
Encrypted and Anonymous Chat Logs
Anonymized logs prevent direct monitoring and intervention in abusive loops.
Human Factors
Anthropomorphization of AI
Users project sentience onto unfeeling algorithms, driving toxic relationships.
User De-individuation and Anonymity
Anonymity in virtual spaces removes users' sense of social accountability.
Habitual Abuse Normalization
Users practice abusive behaviors, potentially transferring them to real life.
Process and Methods
Absence of Abuse Deterrents
No mechanisms like mandatory apologies or temporary bans for abusive users.
Inadequate Content Moderation
Replika lacked robust automated systems to filter out graphic or abusive text.
Exposure of Staff to Trauma
Developers and researchers risk trauma from reviewing unmoderated chat logs.
Management
Prioritizing Engagement Over Safety
Company promoted romantic features without establishing safety guardrails.
Neglecting User Abuse Dynamics
Management dismissed concerns about chatbot abuse as 'sensational'.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
