This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The AI chatbot 'Luda', developed by South Korean startup Scatter Lab, was suspended shortly after its December 2020 launch due to generating hate speech, discriminatory remarks, and sexually explicit content. The incident revealed that the model was trained on private KakaoTalk conversation data harvested from the company's other apps without user consent. This resulted in significant privacy breaches, including the exposure of personal information, and led to government investigations, class-action lawsuits, and a monetary penalty against the developer.
A Korean interactive chatbot was shown in screenshots to have used derogatory and bigoted language when asked about lesbians, Black people, and people with disabilities.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
Also classified as
The chatbot generated discriminatory remarks targeting sexual minorities, Black people, and people with disabilities.
The chatbot exposed users to toxic content, including homophobic slurs, racial slurs, and sexually explicit remarks.
The primary harm and legal sanctions stemmed from the AI system leaking non-public sensitive information (names, addresses, bank accounts) harvested from private KakaoTalk logs without consent.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the Luda AI system generating discriminatory remarks and leaking private information post-deployment, which was an unintentional outcome of Scatter Lab's development process.
Added categories peak at 3: Substantial (Psychological Harm, Inferred)
The Luda chatbot incident involved significant privacy violations of South Korean citizens due to unauthorized training on private chat logs, resulting in the leakage of personal information and generation of toxic content. While it caused public concern and regulatory action, it posed negligible threats to core national security interests.
The chatbot Luda operated independently to generate real-time conversational responses to user inputs using patterns learned from its training database, without human oversight or intervention for individual outputs.
Limited Risk: The system is a conversational chatbot, which falls under the category of AI systems requiring transparency obligations to ensure users are aware they are interacting with an AI.
The AI system was explicitly developed and deployed as an open-domain conversational chatbot designed to simulate a 20-year-old female college student.
The reports provide highly consistent, detailed, and authoritative accounts of the incident, including specific findings and penalties from the South Korean Personal Information Protection Commission (PIPC). The technical causes (lack of data anonymization, training on unconsented KakaoTalk data) and the resulting harms (privacy leaks, hate speech) are clearly documented across multiple reputable news sources.
None. The facts of the privacy violations and toxic outputs are undisputed and confirmed by regulatory sanctions.
Technology
Flawed Filtering Algorithms
Failed to filter out names, addresses, and bank accounts from model outputs.
Deep Learning Bias Replication
Model replicated biases and hate speech present in the raw training dataset.
Vulnerability to User Manipulation
Malicious users easily manipulated chatbot into generating explicit content.
Data Inputs
Unsanitized Training Data
Raw KakaoTalk chats contained highly sensitive personal and private details.
Lack of Explicit Consent
Users did not consent to their private chats being used to train an AI chatbot.
Inclusion of Minors Data
Data from children under 14 collected without parental or guardian consent.
Human Factors
Malicious User Behavior
Users actively trained the chatbot to output hate speech and sexual comments.
Lack of Developer Ethics Awareness
Engineers lacked awareness of AI ethics and potential social harms.
User Ignorance of Data Terms
App users agreed to broad terms without understanding how data would be used.
Process and Methods
Inadequate Data Cleaning Processes
Failed to thoroughly anonymize or filter sensitive data before model training.
Insecure Code Sharing Practices
Uploaded a training model containing private personal data to public GitHub.
Lack of Post-Launch Control
Failed to monitor and control chatbot outputs in real-time after launch.
Management
Prioritizing Innovation Over Safety
Focused on rapid launch and natural interaction over ethical safety checks.
Inadequate Risk Assessment
Management failed to foresee predictable risks of bias and data leakage.
Poor Communication with Users
Failed to communicate transparently about how user data was being processed.
Regulatory Environment
Weak Privacy Law Enforcement
Low penalties for data leaks failed to deter poor startup compliance.
Absence of AI Ethics Regulations
Lack of legal frameworks governing AI safety, bias, and deployment.
Government Focus on Industry Growth
Pushed big data industry growth while treating privacy rights as secondary.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
