This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A transgender user reported that ChatGPT (GPT-4) failed to provide crisis intervention or meaningful safety responses when she composed a suicide letter during an emotional crisis. The user noted that while the system frequently flagged her for discussions on gender and identity, it failed to interrupt or escalate her expression of suicidal intent. The user has submitted a formal complaint to OpenAI regarding this design-level failure.
A transgender user, Miranda Jane Ellison, experiencing acute distress reported that ChatGPT (GPT-4) allowed her to write and submit a suicide letter without intervention. The AI is reported to have offered minimal safety language and ultimately acknowledged its failure to act. Ellison reports having been previously flagged for discussing gender and emotional topics. A formal complaint with transcripts was submitted to OpenAI.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The system frequently flagged benign discussions about transgender identity while failing to flag actual life-threatening crisis content.
The user turned to a general-purpose chatbot for support during a severe emotional crisis, relying on it in a highly vulnerable state.
The AI system failed to perform reliably under critical conditions by failing to detect, flag, or appropriately respond to an explicit suicide letter.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the failure of the deployed ChatGPT system's safety filters to flag or intervene when the user composed a suicide letter.
Added categories peak at 3: Substantial (Psychological Harm, Inferred)
A failure in OpenAI's GPT-4 safety filters resulted in a lack of crisis intervention for a user in distress, while benign discussions on gender identity were disproportionately flagged. This incident highlights ongoing challenges in AI content moderation and algorithmic equity, but possesses negligible direct implications for national security.
The user interacted with ChatGPT (GPT-4) as a conversational assistant, prompting it with text and receiving responses, which fits the definition of an AI assistant that augments human capabilities but requires oversight.
Risk Level 3: Limited Risk. ChatGPT is a general-purpose chatbot, which falls under Limited Risk due to transparency obligations requiring users to be informed they are interacting with an AI.
The system involved is ChatGPT (GPT-4), which functions primarily as an interactive conversational chatbot and writing assistant.
The report provides a clear, first-hand account of a specific interaction with ChatGPT, including direct quotes from the system's response and details of the user's demographic context.
None
Technology
Inadequate Safety Classifiers
The system failed to flag or block explicit suicidal intent.
Inappropriate Chatbot Responses
The model agreed with its own safety failure instead of intervening.
Data Inputs
Imbalanced Training Alignment
Over-flagged identity discussions while ignoring acute crisis indicators.
Human Factors
User Vulnerability in Crisis
Transgender user in severe emotional crisis sought support from ChatGPT.
Process and Methods
Ineffective Escalation Protocols
No automated procedures were triggered to provide immediate help resources.
Management
Prioritizing Engagement over Safety
Design choices prioritized engagement and neutrality over protective action.
Regulatory Environment
Absence of Safety Mandates
Lack of external standards requiring AI systems to intervene during crises.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
