This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The WeChat messaging app's neural machine translation feature was found to translate the neutral Chinese term for 'black foreigner' into a racial slur when used in negative contexts. The issue was identified by a user who received an offensive translation of a colleague's message. Tencent, the developer, acknowledged the error, attributed it to the AI's training on large datasets, and subsequently updated the system to remove the slur.
The Chinese platform WeChat provided an inappropriate and racist English translation for the Chinese term for “black foreigner” in its messaging app.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The translation engine unfairly associated a neutral term for black individuals with a derogatory racial slur specifically in negative contexts, representing racial bias.
The AI system generated and exposed a user to a highly offensive racial slur (hate speech) due to biased associations in its translation algorithm.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by an unexpected translation output from WeChat's neural machine translation system after it was deployed to users.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
An unintentional translation error in WeChat's neural machine translation system translated a neutral Chinese phrase into a racial slur in negative contexts. While causing distress and highlighting algorithmic bias, the incident was quickly patched by Tencent, posing negligible threats to national security.
The translation software operates as a tool, performing a specific, well-defined function of translating text based on direct user input without context retention or autonomous planning.
Minimal or No Risk: The translation software is a general-purpose utility tool integrated into a messaging app, posing low or negligible systemic risk to society under the EU AI Act framework.
The AI system's sole function in this incident was translating text from Chinese to English.
The reports from multiple reputable sources are highly consistent, detailing the exact phrase, the context of the translation error, the user's reaction, and the developer's response and subsequent fix.
None.
Technology
Unconstrained Neural Translation
AI learned and literalized negative associations without output constraints.
Missing Output Filter
System lacked a filter to block translation of known offensive slurs.
Data Inputs
Biased Training Data
AI was trained on text containing racial slurs and negative associations.
Negative Context Association
Training data linked neutral terms with slurs in negative contexts.
Human Factors
No Human Oversight
Complete automation removed human review from the translation process.
Lack of Cross-Cultural Awareness
Developers lacked sensitivity to global racial slurs and dynamics.
Process and Methods
Inadequate Pre-release Testing
Testing failed to evaluate translation outputs under negative contexts.
Reactive Moderation Process
Relied on user complaints rather than proactive safety checks.
Management
Inadequate Risk Assessment
Management failed to assess risks of offensive AI-generated content.
Growth Prioritized Over Safety
Rapid scaling of translation service occurred without safety safeguards.
Regulatory Environment
Asymmetric Censorship Rules
Regulations mandated political censorship but neglected racial slurs.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
