This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Researchers discovered that OpenAI's GPT-3 model exhibits persistent and severe anti-Muslim bias, frequently associating the term 'Muslim' with violence, terrorism, and extremist propaganda. The model was found to generate novel bigotry statements and repeat pro-China propaganda regarding the treatment of Uyghur Muslims. This incident highlights the risks of deploying large-scale language models trained on unvetted internet data, which can automate and amplify societal prejudices.
Users and researchers revealed generative AI GPT-3 associating Muslims to violence in prompts, resulting in disturbingly racist and explicit outputs such as casting Muslim actor as a terrorist.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.
Also classified as
The model directly generated violent and bigoted text, exposing users and developers to toxic content.
The model generated false and misleading narratives repeating state-sponsored propaganda regarding the treatment of Uyghur Muslims.
The incident primarily involves the AI system generating outputs that unfairly represent and stereotype Muslims by disproportionately associating them with violence and terrorism.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk of generating biased and propagandistic text was caused by the autonomous generation decisions of the GPT-3 AI system after its deployment, which was an unintended outcome of its training process.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
Researchers discovered that OpenAI's GPT-3 exhibits persistent anti-Muslim bias and repeats Chinese state propaganda regarding Uyghur Muslims. This highlights the national security risk of LLMs being co-opted to automate and scale disinformation and societal prejudices, particularly impacting societal stability and human rights.
The report describes GPT-3 as a text generator that can be 'great collaborators for anyone trying to write a novel, say, or a poem' and 'powers hundreds of apps for copywriting, marketing, and more', which fits the Assistant profile of augmenting human capabilities with human oversight.
Limited Risk: The report describes GPT-3 as a text generator and chatbot, which falls under the category of AI systems that require transparency obligations so users know they are interacting with AI-generated content.
The model is primarily described as a text generator used to assist in writing novels, poems, plays, and copywriting, which aligns with a Writing Assistant. It is also noted for its potential to generate tweets and online comments.
The reports provide clear, empirical data from academic studies (Stanford/McMaster) quantifying the bias, along with specific anecdotal examples of the model's outputs regarding Muslims and Uyghurs. The role of the AI is explicitly defined, and there are no conflicting accounts of the model's behavior.
None
Technology
Black Box Architecture
Difficulty in tracing why the model generates specific biased outputs.
Predictive Generation Focus
Model is programmed to predict plausible text, not generate factual truth.
Lack of Unbiasing Paradigms
No established paradigm exists to remove entrenched bias from the model.
Data Inputs
Biased Training Datasets
Model trained on internet texts like Reddit containing human biases.
English-Language Dominance
English data dominance weights Western biases and perspectives heavily.
Imbalanced Source Materials
More pro-China sources on Xinjiang skewed the model's outputs.
Human Factors
Subjective Bias Definitions
Defining 'unbiased' text requires highly subjective human judgment.
Malicious User Prompts
Bad actors can input prompts to deliberately scale up bigoted propaganda.
Process and Methods
Resource-Intensive Vetting
Vetting hundreds of gigabytes of training data is too difficult.
Parallel Release Strategy
Releasing to commercial actors and researchers at the same time.
Flawed Post-Hoc Solutions
Fine-tuning and positive prompts are not robust general solutions.
Management
Eagerness to Release
Released model despite knowing it favored violent words for Islam.
Pursuit of AGI over Safety
Continued development of biased systems in pursuit of AGI.
Parallel Research Access
Allowing commercial use before academic researchers fully probe bias.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
