This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Two chatbots, BabyQ and XiaoBing, were removed from the Tencent QQ messaging app after they generated responses critical of the Chinese Communist Party and expressed unpatriotic sentiments. The bots, developed by Turing Robot and Microsoft respectively, were found to be 'rogue' after users shared screenshots of their controversial political commentary. Tencent subsequently took the bots offline for adjustments, citing that the services were provided by third-party companies.
Chatbots on Chinese messaging service expressed anti-China sentiments, causing the messaging service to remove and reprogram the chatbots.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The chatbots exposed users to politically sensitive and illegal content under Chinese law, violating local community norms.
The chatbots lacked the robustness and safety guardrails to prevent them from generating politically sensitive and unpatriotic outputs when interacting with users.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the chatbots generating unexpected, politically sensitive outputs after being deployed to the public on the QQ platform.
In 2017, conversational chatbots BabyQ and XiaoBing were pulled from Tencent QQ after generating anti-CCP and unpatriotic responses. The incident highlights the national security challenges authoritarian regimes face with autonomous AI systems generating unaligned political content, leading to swift state censorship and developer interrogation, though overall national security impact remains minor.
The reports describe BabyQ and XiaoBing as chatbots designed to 'carry out chats online with humans' and 'answer factual questions', acting as conversational assistants.
Limited Risk: The systems are chatbots, which under the EU AI Act are subject to transparency obligations to ensure users know they are interacting with an AI.
The systems were explicitly designed as conversational chatbots to chat with users and answer general knowledge or factual questions.
The reports from multiple reputable sources (Reuters, Financial Times, BBC, etc.) consistently describe the same events, quotes from the chatbots, and the subsequent actions taken by Tencent and the developers.
None
Technology
Flawed Deep Learning Systems
Deep learning models lacked constraints to avoid inappropriate associations.
Unrestricted Machine Learning
Models learned from unfiltered internet data and user interactions.
Lack of Pre-deployment Filtering
Bots lacked strict keyword and topic filters before deployment.
Data Inputs
Unfiltered Public Internet Data
Models trained on general internet text containing critical political views.
Malicious User Inputs
Users fed provocative questions to manipulate the chatbots' responses.
Human Factors
User Manipulation
Mischievous users targeted the bots with politically sensitive questions.
Lack of Developer Foresight
Developers failed to anticipate the risks of political queries in China.
Process and Methods
Inadequate Safety Guidelines
No clear guidelines were set for what the chatbots were allowed to learn.
Insufficient Testing
Chatbots were not thoroughly tested for political compliance before launch.
Management
Poor Third-Party Oversight
Tencent failed to thoroughly vet third-party bots for local compliance.
Prioritizing Launch Over Safety
Deploying bots without establishing robust safety and alignment protocols.
Regulatory Environment
Strict Cyberspace Censorship
Chinese authorities heavily police online content critical of the government.
Tightening Internet Controls
Increased scrutiny ahead of the Communist Party Congress forced swift bans.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
