This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Google Lens's camera-based translation feature mistranslated a Korean book title into an offensive slur. The error occurred because the model lacked the context to distinguish between a formal book title and slang usage of the Korean word for 'that', which is also used as a term for male genitalia in internet message boards. This incident highlights the challenges of model generalization and the risks of training on internet communications without sufficient contextual awareness.
A book title by Korea’s first minister of culture was mistranslated into an offensive phrase by Google Lens’s camera-based translation feature allegedly due to its training on internet communications and a lack of context.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The system failed to perform robustly under distributional shift, failing to understand the context of a formal book title versus internet slang.
The AI system generated and exposed the user to a highly offensive and profane sexual slur, violating community norms regarding appropriate content.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The mistranslation was caused by the Google Lens AI system's failure to generalize context post-deployment, which was an unintended outcome of its translation goal.
A consumer-facing translation tool (Google Lens) mistranslated a Korean book title into an offensive English slur due to training data bias and lack of context. The incident represents a minor post-deployment software bug with negligible implications for national security.
The AI system acts as a tool because it performs a specific, well-defined function (translating text from an image) on direct input (the photo of the book cover) without maintaining an ongoing state or planning.
Minimal or No Risk: The system is a general-purpose translation tool used for everyday text translation, which poses low or negligible risk to safety or fundamental rights under the EU AI Act.
The primary goal of the Google Lens feature in this context was to translate text from an image (the book cover) into English.
The report provides clear, direct evidence of the translation error, including side-by-side images of the book cover and the resulting translation. The linguistic explanation for why the error occurred is detailed and logical, leaving little ambiguity about the nature of the incident.
None
Technology
Generalization Challenge
Models struggle to perform beyond engineered environments.
Lack of Contextual Awareness
AI cannot resolve subtext or differentiate literal from slang.
Data Inputs
Internet Slang Training Data
Training on web forums leads to offensive slang translations.
Lack of Labeled Context
Missing metadata about text genre or source style.
Human Factors
Overreliance on Automation
Assuming machine translation is robust across low-resource languages.
Process and Methods
Lack of Translation Validation
No systematic verification of output quality before scaling.
Missing Feedback Loop
No immediate way for users to report or correct errors.
No Source Text Comparison
Failure to provide links back to original untranslated text.
Management
Premature Feature Scaling
Prioritizing rapid language rollout over robust verification.
Lack of Human-in-the-Loop
Deploying translation without community review oversight.
