This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The AI-powered game AI Dungeon, which utilized OpenAI's GPT-3, was found to generate sexually explicit content involving children when prompted by users. This led to a conflict between the developer, Latitude, and OpenAI regarding content moderation. Additionally, a security vulnerability in the game resulted in the exposure of hundreds of thousands of private user-generated stories to the public.
Latitude's GPT-3-powered game AI Dungeon was reportedly abused by some players who manipulated its AI to generate sexually explicit stories involving children.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
A security vulnerability exposed hundreds of thousands of private, user-generated stories to the public.
The GPT-3 model generated stories depicting sexual encounters involving children, which constitutes exposure to toxic content and child sexual abuse material.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The generation of sexually explicit content involving children was an unexpected and unintentional output of the GPT-3 model post-deployment, triggered by user prompts.
Added categories peak at 2: Minor (Child Sexual Exploitation & Abuse (CSEA), Direct)
An early commercial LLM deployment (AI Dungeon) was abused by users to generate child-exploitation-adjacent text, coupled with a security vulnerability leaking user data. While presenting serious ethical and privacy concerns, the national security impact is negligible to minor, with no threat to physical infrastructure, intelligence, or state sovereignty.
The report describes the game as a 'choose-your-own adventure game' representing 'a new form of human-machine collaboration' where 'when a player typed out the action or dialog... algorithms would craft the next phase of their personalized, unpredictable adventure.' This fits the Assistant profile as it augments human creative writing under human direction.
Limited Risk: The system is a generative AI text creator and interactive game, which falls under transparency obligations for AI-generated content.
The AI system was primarily used to generate personalized, unpredictable adventure stories for an online game, acting as a collaborative writing assistant for players.
The report provides clear, detailed accounts of the AI system used (GPT-3), the developer (Latitude), the specific safety failures (generation of child sexual content and a security vulnerability leaking stories), and the resulting user backlash. The facts are well-documented with quotes from both OpenAI's CEO and Latitude's spokespersons.
The incident could be viewed primarily as a software security failure rather than an AI safety failure, though the core conflict stemmed from the AI's generation of toxic content.
Technology
Unpredictable Text Generation
GPT models generate text based on patterns without semantic understanding.
Oversensitive Algorithmic Filters
Filters blocked benign phrases like '8-year-old laptop' causing user anger.
Data Inputs
Unfiltered Training Data
Models trained on internet text containing toxic and inappropriate content.
User Prompt Manipulation
Users input creative prompts to steer the AI into generating NSFW stories.
Human Factors
User Backlash Over Privacy
Players revolted against manual reviews of their private fictional stories.
Intentional Policy Evasion
Some players actively tried to generate prohibited child sexual content.
Process and Methods
Inadequate Content Moderation
Latitude relied on basic word filters that were easily bypassed or too strict.
Intrusive Manual Review Plan
Reading flagged private stories violated user trust and privacy expectations.
Management
Prioritizing Growth Over Safety
Latitude offered unconstrained access to GPT-3 without robust guardrails.
Upstream Pressure from OpenAI
OpenAI demanded immediate action after discovering policy violations.
Regulatory Environment
Lack of AI Safety Standards
No industry-wide regulations governed the deployment of generative models.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
