This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
OpenAI's DALL-E 2 model was found to exhibit significant gender and racial biases, reinforcing societal stereotypes in its generated images. Researchers and internal audits confirmed that the model disproportionately associated certain professions with specific genders and races. In response, OpenAI implemented system-level prompt modifications to increase diversity, a move criticized by some experts as a non-transparent 'sticking plaster' that fails to address the underlying cultural biases in the training data.
Developers of OpenAI's DALL-E 2 cited risks of the model, varying from misuse as disinformation and explicit content generation, to gender and racial bias.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.
Also classified as
The reports highlight concerns that the model's efficiency in generating images could displace jobs for designers, photographers, and artists.
OpenAI's covert modification of user prompts lacked transparency, drawing criticism from researchers regarding the secrecy of their bias mitigation methods.
The DALL-E 2 model generated images that reinforced harmful racial and gender stereotypes, unfairly representing various demographic groups in professional roles.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The biased image generation is an unintentional outcome of the AI system's training, occurring post-deployment during its public preview phase.
Added categories peak at 2: Minor (Epistemic Harm, Indirect)
The incident involved OpenAI's DALL-E 2 model exhibiting racial and gender biases during its preview phase, and the subsequent implementation of covert prompt modifications to address the issue. The national security impact is minor, primarily touching on societal stability and human rights through the potential reinforcement of stereotypes, with no direct threats to critical infrastructure, sovereignty, or intelligence.
DALL-E 2 operates as an Assistant. It takes text prompts from users and generates corresponding images to augment human creative workflows, requiring direct human input and oversight for every generation.
Limited Risk: The system is an AI-generated content tool (text-to-image generator). Under the EU AI Act, 'AI-generated content such as deepfakes' and systems generating synthetic content fall under Limited Risk, requiring transparency obligations to inform users that the content is AI-generated.
The primary function of DALL-E 2 is to generate images from textual descriptions, which also serves the secondary goal of visual art generation.
The reports provide detailed, multi-perspective accounts of DALL-E 2's biases, OpenAI's technical mitigations (prompt injection), and the system card details. The facts are well-documented by both independent researchers and OpenAI's own public disclosures.
Some might view the prompt modification as a successful mitigation rather than an incident, but the lack of transparency and the underlying bias make it a safety/fairness incident.
Technology
Inherited Dataset Bias
Model inherits prejudices from training data scraped from the internet.
Covert Prompt Modification
System covertly injects words to alter prompts without user knowledge.
Reference Collisions
Single words with multiple meanings cause unintended explicit generations.
Data Inputs
Biased Training Data
Training data overrepresents white-passing and male-passing demographics.
Flawed Training Data Filters
Filtering sexual content inadvertently reduced the generation of women.
Missing Local Contexts
Training data lacks diverse international and local cultural representations.
Human Factors
Target Harassment Risks
Users can exploit image modification capabilities to harass individuals.
Bypassing Safe Filters
Users use visual synonyms and coded language to bypass prompt safety filters.
Trust in Fake Evidence
Believable synthetic images reduce public trust in overall information systems.
Process and Methods
Lack of Transparency
Secrecy in model development prevents public assessment of bias mitigations.
Missing Bias Standards
The AI industry lacks standard metrics or norms to measure and solve bias.
Imperfect Prompt Classifiers
Prompt filters fail when neutral words are combined in harmful ways.
Management
Competitive Pressures
Market competition motivates releasing biased models before resolving safety.
Sticking Plaster Technical Fix
Management chose a superficial fix instead of systemic cultural change.
Exclusive Early Access
Releasing to a selected network creates unfair first-mover advantages.
Regulatory Environment
Absence of Hard Regulation
No legislative frameworks enforce safety standards in AI deployment.
Relying on Soft Regulation
Voluntary best practices fail to create repeatable, industry-wide standards.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
