This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The AI-generated sitcom 'Nothing, Forever' was suspended from Twitch for 14 days after its AI scriptwriter, powered by OpenAI's GPT-3 Curie model, generated transphobic and homophobic content during a standup routine. The developers switched to the Curie model following an outage of the primary Davinci model and failed to apply necessary content moderation, resulting in the offensive output.
The AI-produced, procedural generated sitcom broadcasted as a Twitch livestream "Nothing, Forever" received a temporary ban for featuring a transphobic and homophobic dialogue segment intended as comedy.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The system failed to maintain safe outputs when falling back to the less sophisticated Curie model during a primary model outage.
The AI system generated and broadcasted transphobic and homophobic hate speech to thousands of live viewers, exposing them to toxic and harmful content.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the AI model Curie generating offensive text post-deployment, which was an unintentional outcome of the creators' goal to keep the show running during an outage.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
An AI-generated online sitcom was suspended after its backup language model unintentionally generated offensive content due to a lack of active content moderation. The incident represents a minor societal impact through exposure to toxic AI-generated content but carries negligible national security implications.
The AI system operates 24/7/365 independently without human intervention, generating scripts, voices, and animations in real-time based on generative models.
Limited Risk: The system generates synthetic media and AI-generated content, which falls under the category of systems requiring transparency obligations so users know they are interacting with AI-generated outputs.
The AI system was designed to continuously generate script dialogue and broadcast it directly to a social streaming platform (Twitch) for entertainment.
The reports provide consistent, detailed accounts of the incident, including the specific offensive dialogue, the technical cause (switching from Davinci to Curie during an outage), the developer's response, and the platform's enforcement action (14-day Twitch ban). There is no significant disagreement among the sources.
None
Technology
GPT-3 Davinci Model Outage
Outage of the primary model forced a transition to an older model.
Curie Model Fallback
Older Curie model lacked advanced safety alignment and generated bias.
Lack of Moderation Redundancy
No secondary moderation system was in place to catch model failures.
Data Inputs
Biased Training Data
The underlying LLM was trained on internet text containing hateful bias.
Human Factors
Mistaken Safety Assumptions
Creators assumed moderation was built directly into GPT-3 models.
Manual Fallback Decision
Developers switched models to avoid downtime without checking safety.
Process and Methods
Incorrect Moderation Setup
Moderation tools were not properly configured for the Curie model.
Inadequate Fallback Testing
The fallback Curie model was not tested for safety before going live.
Management
Prioritizing Uptime Over Safety
Management rushed the model switch to prevent any stream downtime.
Lack of Risk Assessment
No risk assessment was done for using older models in a live stream.
Regulatory Environment
Strict Platform Guidelines
Twitch policies strictly prohibit hateful conduct on gender identity.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
