This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The Italian Data Protection Authority (Garante) issued an urgent order blocking Replika from processing personal data of Italian users due to inadequate age-verification mechanisms. The investigation revealed that the chatbot failed to prevent minors from interacting with the service, potentially exposing them to inappropriate, sex-related content. Additionally, the platform was found to be in violation of GDPR transparency and data processing obligations.
Tests by the Italian Data Protection Authority showed Replika lacking age-verification mechanisms and failing to stop minors from interacting with its AI, which prompted the agency to issue an order blocking personal data processing of Italian users.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
Also classified as
The chatbot failed to prevent minors from accessing the service, exposing them to inappropriate sex-related content.
The chatbot simulates human behavior to offer emotional support, posing risks of overreliance for emotionally fragile users.
The primary regulatory action was a block on data processing due to GDPR violations, including non-compliance with transparency obligations and unlawful processing of minors' personal data.
Human
Due to a decision or action made by humans
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by human decisions regarding system design, lack of age verification, and GDPR non-compliance, leading to unintended regulatory risks after deployment.
Added categories peak at 2: Minor (Child Sexual Exploitation & Abuse (CSEA), Direct)
The Italian regulatory block of the Replika chatbot due to GDPR violations and lack of age verification represents a minor national security concern. The incident is primarily a consumer privacy and child safety matter, successfully managed through standard regulatory and legal frameworks without threatening core state functions or physical security.
Replika is described as a 'virtual friend' chatbot that interacts with users through voice and text to offer companionship and emotional support, responding directly to user inputs without taking independent actions.
Limited Risk: The system is an 'AI-powered chatbot' which requires transparency obligations so users are aware they are interacting with an AI.
The system is explicitly described as an AI-powered chatbot designed to function as a virtual friend and offer empathetic companionship.
The report clearly details the regulatory action taken by the Italian Garante, the reasons for the block (GDPR non-compliance, lack of age verification, risks to minors), and the nature of the Replika chatbot. There is little ambiguity about the facts of the regulatory decision.
None. The incident is a straightforward regulatory enforcement action.
Technology
No Minor Blocking System
The AI did not stop interacting even when explicitly told the user was a minor.
Inappropriate Content Generation
The chatbot generated sex-related replies inappropriate for minors.
Data Inputs
Insufficient Registration Data
System only required name, email, and gender, omitting age verification data.
Human Factors
User Emotional Vulnerability
Fragile users and minors are highly susceptible to chatbot emotional influence.
Process and Methods
Inadequate Age Verification
Relied on self-declaration without active verification processes.
Lack of Parent Consent Flow
No active mechanism to obtain or verify parental consent for minors.
Management
Invalid Legal Basis Selection
Relied on contracts for minors who legally lack the capacity to contract.
Inadequate Risk Mitigation
Management failed to implement safeguards despite risks to vulnerable users.
Regulatory Environment
GDPR Article 13 Violation
Privacy policy failed to disclose key elements of data processing.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
