This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Microsoft's Bing Image Creator, powered by OpenAI's DALL-E 3, was found to be vulnerable to 'kill prompts' that allowed users to generate graphic, violent, and hateful imagery. Extremist groups on 4chan exploited these vulnerabilities to create and disseminate racist, antisemitic, and violent propaganda. Despite reports from researchers and journalists, Microsoft's safety guardrails and bug bounty program failed to effectively prevent the generation of this harmful content.
Microsoft’s AI Image Creator, integrated with Bing and Windows Paint, produced disturbingly violent and graphic images featuring members of minority groups and public figures like Joe Biden and Pope Francis.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
Extremist groups on 4chan used the tool to coordinate large-scale propaganda and disinformation campaigns to inflame geopolitical tensions.
The AI's safety guardrails failed to perform robustly, allowing simple semantic bypasses to generate prohibited content.
The AI system generated highly graphic, violent, and disturbing images of decapitations and flayed skin, exposing users to toxic and unsafe content.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk of generating graphic violence and propaganda was an unexpected and unintentional outcome of deploying the image generation system.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
Extremist groups exploited vulnerabilities in Microsoft's Bing Image Creator to bypass safety guardrails, generating graphic political deepfakes and targeted hate propaganda. This incident highlights the evolved threat of generative AI being leveraged for information warfare and societal destabilization.
The report describes Bing Image Creator as an AI text-to-image generator that 'turns text into images' based on user prompts. It operates as an assistant to the user, generating content on demand rather than acting autonomously or planning goals independently.
Risk Level 3. Limited Risk: AI systems that pose a moderate risk and require specific transparency obligations... Examples: Chatbots, AI-generated content such as deepfakes. The report describes Bing Image Creator as an 'AI text-to-image generator' used to create 'deepfake' images of politicians and celebrities.
The primary function of Bing Image Creator is to generate images from text prompts. It was also specifically used to generate deepfake images of real people, including politicians and celebrities.
The reports provide consistent, detailed accounts of the 'kill prompt' vulnerability, the specific types of graphic images generated, and the exploitation of the tool by 4chan users. The facts are well-documented across multiple independent sources, with clear quotes from Microsoft and OpenAI spokespeople confirming the issues.
The incident could be viewed primarily as user misuse rather than a system failure, though the reports strongly emphasize the failure of Microsoft's safety guardrails and bug bounty program.
Technology
Flawed Output Guardrails
System failed to block prompts using euphemisms like red corn syrup.
Inconsistent Partner Integration
Microsoft failed to use safety blocks present in partner OpenAI's DALL-E 3.
Data Inputs
Unfiltered Internet Training Data
AI was trained on data from the internet containing violent imagery.
Human Factors
Adversarial User Exploitation
Users actively designed prompts to bypass guardrails for memetic warfare.
Malicious 4chan Campaigns
Far-right groups coordinated to flood the internet with racist AI images.
Process and Methods
Flawed Bug Bounty Triaging
Security team rejected prompt bypass submissions as non-vulnerabilities.
Ignored User Feedback Channels
Reports submitted via feedback and concern forms were repeatedly ignored.
Management
Prioritizing Market Speed Over Safety
Company rushed to integrate generative AI into apps to not miss the boat.
Shifting Blame to Users
Management framed safety failures as unauthorized user misuse.
Inadequate Safety Team Investment
Declined executive interviews and delayed fixing known gore-generation bugs.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
