This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Delphi, a research prototype AI from the Allen Institute for AI, was released to provide moral judgments on ethical dilemmas. The system quickly went viral for generating offensive, racist, and homophobic content, as well as condoning harmful behaviors based on user-provided framing. The incident highlights the risks of deploying models trained on internet-scraped data without sufficient safeguards for moral reasoning.
A publicly accessible research model that was trained via Reddit threads showed racially biased advice on moral dilemmas, allegedly demonstrating limitations of language-based models trained on moral judgments.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.
Also classified as
The system exposed users to toxic content by condoning genocide, suggesting a user should die, and outputting white supremacist slogans.
The AI system generated explicitly racist and homophobic moral judgments, representing unequal and discriminatory treatment of demographic groups.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The offensive and biased moral judgments were generated by the Ask Delphi AI system after its public deployment, representing an unexpected outcome of its training.
Added categories peak at 3: Substantial (Psychological Harm, Inferred)
The Ask Delphi research prototype generated offensive, biased, and toxic moral judgments when prompted by public users. While highlighting critical challenges in AI alignment, training data bias, and safety guardrails, the incident poses negligible direct threats to national security, critical infrastructure, or state sovereignty.
The AI system operates as a simple web-based tool that takes a single text input from a user and outputs a static moral judgment without maintaining an ongoing dialogue, planning, or taking autonomous actions.
Limited Risk: The system is a public chatbot and AI-generated content generator, which falls under Risk Level 3 and is subject to transparency obligations to ensure users know they are interacting with an AI.
The AI system was deployed as an interactive web-based chatbot designed to model and output human moral judgments based on user-submitted scenarios.
The reports from multiple reputable media outlets provide consistent, detailed accounts of the AI system's deployment, training methodology, specific offensive outputs, and the developers' subsequent updates and disclaimers.
The incident could be interpreted primarily as a failure of robustness (7.3) rather than discrimination (1.1), but the explicit group-based biases support the discrimination classification under the precedence ladder.
Technology
Lack of Common-Sense Understanding
Large language models lack actual understanding of human morality and concepts.
Extreme Sensitivity to Phrasing
Minor verbal triggers easily flip judgments from condemnation to approval.
Absence of Emotional Capacity
Neural networks lack the emotional foundation necessary for moral reasoning.
Data Inputs
Biased Training Data Sources
Prompts scraped from Reddit subreddits contain inherent societal biases.
Subjective Crowdworker Labeling
MTurk workers labeled data based on subjective, localized American norms.
Human Factors
User Tendency to Anthropomorphize
Users view the AI as a mystical, objective authority rather than a tool.
Lack of Public AI Literacy
Users do not understand the technical context or limitations of the model.
Malicious Prompt Engineering
Users actively game the system with phrasing to force abhorrent answers.
Process and Methods
Inadequate Pre-Release Testing
Public quickly discovered offensive outputs that internal testing missed.
Flawed Comparison Tool Design
Feature comparing two situations enabled highly bigoted output generation.
Lack of Safe Default Fallbacks
System lacked rule-based overrides for extreme prompts like suicide or war.
Management
Prioritizing Virality over Safety
Features like Twitter sharing were built before safety guardrails were set.
Misleading Initial Framing
Presenting the system as an oracle rather than a highly flawed prototype.
Regulatory Environment
Lack of Legal Liability for AI
Absence of legal frameworks holding developers accountable for AI outputs.
