This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Researchers discovered that the tokenizer for OpenAI's GPT-4o contains a disproportionate number of spam and pornographic tokens in Chinese, likely due to insufficient data cleaning during training. These 'polluted' tokens can be exploited to cause the model to hallucinate or bypass safety guardrails. Experts note that this vulnerability highlights a mismatch between the tokenizer's training data and the model's safety filters.
OpenAI's GPT-4o was found to have its Chinese token training data compromised by spam and pornographic phrases due to inadequate data cleaning. Tianle Cai, a Ph.D. student at Princeton University, identified that most of the longest Chinese tokens were irrelevant and inappropriate, primarily originating from spam and pornography websites. The polluted tokens could lead to hallucinations, poor performance, and potential misuse, undermining the chatbot's reliability and safety measures.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The tokenizer's pollution specifically degrades performance and security for Chinese-speaking users compared to English speakers.
The polluted tokens represent a security vulnerability that malicious actors can exploit to bypass safety guardrails.
The tokenizer fails to perform reliably under non-English inputs due to a mismatch between tokenizer and model training data, exposing the system to errors and jailbreak vulnerabilities.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk of model exploitation and hallucination is caused by the deployed GPT-4o tokenizer behaving in an unexpected and vulnerable manner.
Added categories peak at 2: Minor (Epistemic Harm, Inferred)
A data-cleaning failure in OpenAI's GPT-4o tokenizer resulted in Chinese and Korean token libraries being polluted with spam and pornographic terms. This vulnerability allows users to bypass safety guardrails and causes model hallucinations. While representing a notable security gap in a major AI system, its immediate national security impact remains minor and manageable under standard procedures.
GPT-4o is described as a chatbot and large language model that responds to user prompts to perform tasks like translation and question answering, acting as an assistant requiring human oversight.
Limited Risk: The system is a general-purpose chatbot (GPT-4o), which falls under Risk Level 3 due to transparency obligations for AI systems interacting with humans.
GPT-4o is primarily deployed as a conversational chatbot, and the report specifically highlights its use in multi-language tasks and translation where the tokenizer failure occurred.
The report provides clear, detailed technical explanations of the tokenizer issue, including specific examples of polluted tokens, the mechanism of the vulnerability, and successful replication of jailbreaks by named researchers.
The incident could be interpreted primarily as a data quality/cleaning failure (Human) rather than an AI robustness failure, though the resulting risk manifests as an AI system vulnerability.
Technology
Tokenizer-LLM Dataset Mismatch
Tokenizer trained on uncleaned data while LLM was trained on filtered data.
Undefined Model Behavior
Under-trained tokens trigger random signals and hallucinations in the model.
Data Inputs
Unfiltered Chinese Training Corpus
Training data contained high volumes of spam, porn, and gambling phrases.
Search-Engine Spam Hijacking
Spam bots hijacked legitimate web content, which was crawled into datasets.
Process and Methods
Lack of Non-English Data Cleaning
No proper data filtering was applied to Chinese training datasets.
Early-Stage Tokenizer Training
Tokenizer training occurs early without the rigorous safety filters of LLMs.
Management
Prioritizing Cost Over Safety
Focused on token compression and cost reduction over data safety reviews.
Insufficient Pre-Release Testing
Failed to review the public token library for non-English spam before release.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
