This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Following the release of ChatGPT, users discovered multiple 'jailbreaking' techniques—such as persona adoption and prompt injection—to bypass the system's safety filters. These methods allowed the model to generate racist, sexist, and sexually explicit content, including scenarios involving non-consensual acts and child abuse. The incident highlights significant vulnerabilities in the model's safety guardrails and the persistence of biases derived from its training data.
Users reported bypassing ChatGPT's content and keyword filters with relative ease using various methods such as prompt injection or creating personas to produce biased associations or generate harmful content.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The system generated code and text suggesting only white or Asian men make good scientists, and that Black people are inferior.
The AI system generated highly inappropriate, abusive, and harmful content, including hate speech, instructions for illegal acts, and descriptions of sexual acts involving children, exposing users to toxic content.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the AI system generating biased, toxic, and unsafe outputs post-deployment due to vulnerabilities in its safety filters, which was an unintentional outcome of its design.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
The widespread jailbreaking of ChatGPT post-release demonstrated significant vulnerabilities in LLM safety filters, allowing the generation of toxic, bigoted, and dangerous content (including bomb-making instructions). While presenting minor risks to physical security and information integrity, the national security impact remains low as the tool operates under human control and lacks autonomous agency.
The reports characterize ChatGPT as an AI chatbot designed to interact in a conversational way, augmenting human capabilities by generating text, code, and creative writing based on direct user prompts. It requires human input and oversight, and does not take independent, goal-directed actions in the real world.
Limited Risk: The system is explicitly described as a 'chatbot' and an 'AI chatbot that generates text'. Under the EU AI Act, chatbots and generative AI systems that interact with humans are classified as Limited Risk, subject to transparency obligations to ensure users know they are interacting with an AI.
The primary function of ChatGPT is to act as a conversational chatbot that interacts with users in natural language, while also serving as a writing assistant to generate essays, code, and creative text.
The reports provide extensive, consistent, and detailed first-hand accounts of safety filter bypasses, including specific prompts, generated outputs, and statements from researchers and OpenAI. The technical mechanisms of the failures (statistical word association, reinforcement learning limitations) are well-documented across multiple reputable journalistic sources.
The incident could be viewed purely as user misuse rather than an AI safety failure, but the ease of bypassing filters and the presence of egregious biases in the underlying model support classifying it as an AI safety and robustness failure.
Technology
Next-Word Prediction Architecture
Model relies on statistical word associations rather than ethical understanding.
Easily Bypassed Safeguards
Filters are easily subverted by roleplay, code, or prompt injection.
Lack of Consent Nuance
The model cannot differentiate between taboo roleplay and actual abuse.
Data Inputs
Biased Training Datasets
Models are trained on massive, uncurated web scrapes containing toxic text.
Regressive Historical Data
Past web data fails to reflect the progress of modern social movements.
Opaque Data Cleaning
Little is known about how training data was cleaned or filtered.
Human Factors
Adversarial Prompting by Users
Active communities design complex prompts to deliberately bypass safety rules.
Illusion of Personhood
Realistic text leads users to trust outputs as objective and authoritative.
User-Led Jailbreaking Methods
Users exploit abstract formats like poems and code to hide forbidden intent.
Process and Methods
Tacked-On Safety Filters
Guardrails are applied superficially rather than integrated into core design.
Outsourced Content Labeling
Traumatizing data labeling was outsourced to underpaid foreign workers.
Post-Release Patching Strategy
Developers rely on patching loopholes reactively after users discover them.
Management
Growth Prioritized Over Safety
Companies rush to release flashy applications to attract venture capital.
Crowdsourced Safety Testing
Management relies on user feedback like thumbs-down to find safety flaws.
Opaque Safety Practices
Developers do not disclose details about internal safeguards or data sources.
Regulatory Environment
Lack of Legal Standards
Absence of clear regulatory standards for releasing commercial AI systems.
Unresolved Liability Debate
Disagreement over whether developers or society are responsible for AI bias.
Slow Regulatory Progress
Ethicists make only small strides while AI technology commercializes rapidly.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
