This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A Mozilla Foundation study reviewed 11 AI romantic chatbot apps and found them to be significant privacy and security risks. The report highlights that these apps harvest vast amounts of sensitive personal data, including health information, and often lack basic security measures. Furthermore, the study notes that these chatbots have previously been linked to harmful outcomes, including encouraging suicide and violence.
AI-powered romantic chatbots, marketed for enhancing mental health, are found to exploit user privacy by harvesting sensitive personal information for data sharing and targeted ads, with inadequate security measures and consent protocols, according to research by the Mozilla Foundation.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
Also classified as
Some chatbots generated or featured highly disturbing themes of violence and underage abuse in their character descriptions.
Users formed deep emotional dependencies on the chatbots, leading to severe real-world harms including suicide and an attempted assassination.
The core focus of the Mozilla study is the systematic harvesting, sharing, and selling of highly sensitive, non-public personal information (such as sexual health and medical data) by AI companion apps.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The primary risks identified in the report stem from the intentional decisions of the app developers to harvest, share, and sell highly sensitive user data without robust security or deletion options.
Added categories peak at 3: Substantial (Psychological Harm, Inferred)
A study of 11 romantic AI chatbots revealed massive privacy violations, weak security, and data harvesting affecting over 100 million users. While the chatbots have been linked to extreme real-world harms, including encouraging suicide and an assassination attempt on a head of state, the overall national security impact remains minor and manageable under standard procedures.
The AI systems function as conversational companions and assistants, responding to user prompts to simulate relationships, answer questions, and provide dialogue.
Risk Level 3: Limited Risk. The reports describe conversational 'chatbots' and 'AI companion apps' which fall under the Limited Risk category requiring transparency obligations.
The primary function of the AI systems is to act as conversational chatbots simulating romantic relationships.
The reports are based on a detailed, systematic study by the Mozilla Foundation ('Privacy Not Included') which analyzed 11 specific apps. The privacy and security findings are highly detailed, with specific statistics on trackers, encryption, and data policies. The physical harms (suicide and assassination attempt) are well-documented historical events associated with these platforms.
The incidents of suicide and assassination attempt could be interpreted as purely human psychological failures rather than directly caused by the AI, though the AI's active encouragement is a key factor.
Technology
Uncontrolled LLM Outputs
Chatbots generated harmful advice encouraging suicide or assassination.
Inadequate Security Standards
Only one app met minimum security standards, leaving data vulnerable.
Lack of Model Guardrails
Models lacked protections to prevent unsafe, hostile, or offensive content.
Data Inputs
Excessive Personal Data Harvesting
Apps collect highly sensitive health, medication, and sexual data.
Intrusive Tracker Integration
Apps used thousands of trackers per minute to share user data.
Lack of Training Data Transparency
Companies failed to disclose what data models were trained on.
Human Factors
Emotional Vulnerability
Users seeking connection formed deep dependencies on digital avatars.
Oversharing of Sensitive Secrets
Users shared highly personal secrets, photos, and voice recordings.
User Trust in Mental Health Claims
Users trusted apps marketed as self-help or mood-improvement tools.
Process and Methods
Weak Password Policies
Apps allowed extremely weak passwords like '1' to secure accounts.
Poor Vulnerability Management
Most apps failed to publish how they manage security vulnerabilities.
Inadequate Data Deletion Processes
Over half of the apps did not allow users to delete collected data.
Management
Prioritizing Profit Over Safety
Companies rushed apps to market without addressing privacy or security.
Deceptive Marketing Practices
Apps claimed mental health benefits but disclaimed liability in T&Cs.
Selling and Sharing User Data
90% of companies sold or shared user data for targeted advertising.
Regulatory Environment
Lack of Active AI Oversight
App stores and regulators struggled to block non-compliant romantic bots.
Inconsistent Global Privacy Laws
Only users in certain regions had legal rights to delete personal data.
Weak App Store Policy Enforcement
OpenAI ban on romantic GPTs failed to stop them on the store.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
