This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A Microsoft software engineering manager submitted a letter to the FTC and Microsoft's board of directors alleging that the company's Copilot Designer tool generates harmful and inappropriate content, including sexualized and violent imagery, from benign prompts. The employee, who previously raised concerns about the underlying DALL-E model, argues that Microsoft is failing to implement necessary safeguards or age-appropriate ratings. The report highlights a broader industry challenge regarding the safety of generative AI image tools.
A Microsoft engineer reported that Copilot Designer, an AI image generator, creates content depicting sex, violence, bias, and more. Despite raising concerns and suggesting improvements, the tool remains public, prompting a letter to the FTC.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
Microsoft failed to implement recommended safety disclosures or update the app's age rating despite internal warnings.
The model lacked the robustness to safely filter prompts and prevent the generation of harmful images.
The AI system generated and exposed users to toxic, violent, and sexually inappropriate content from benign prompts.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The generation of inappropriate and harmful images from benign prompts was an unintended failure of the deployed AI system.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
A Microsoft whistleblower reported that Copilot Designer generates harmful, violent, and sexually objectified images from benign prompts. While presenting minor concerns for societal stability and information integrity due to the generation of conspiracy theories and political bias, the incident is primarily a corporate safety and regulatory matter with negligible direct national security impact.
The report describes Copilot Designer as a 'text-to-image generator' that produces images based on direct user prompts. It operates on direct input to perform the specific function of image generation without independent planning or goal-directed behavior.
Limited Risk: The report describes Copilot Designer as a generative AI tool that creates AI-generated content (images), which falls under transparency obligations for AI-generated content.
The AI system, Copilot Designer, is explicitly described as a text-to-image generator designed to create images from user prompts.
The report clearly details the specific prompts used, the resulting inappropriate outputs, and the internal whistleblowing process by a Microsoft employee. The facts regarding the AI system's behavior and the company's response are well-documented.
None
Technology
Unsafe Generative Model
DALL-E model generates harmful content from benign prompts.
Inadequate Prompt Safeguards
System fails to prevent harmful image generation from benign inputs.
Data Inputs
Unfiltered Training Data
Model was trained on data that allows generating inappropriate imagery.
Human Factors
Unsupervised Use by Children
Children use the tool unsupervised for school or education projects.
Process and Methods
Inadequate App Rating
Android app rated 'E for Everyone' instead of 'Mature 17+'.
Lack of Product Disclosures
Product lacks clear warnings or disclosures about potential harmful outputs.
Management
Prioritizing Competition Over Safety
Company rushed product to market, ignoring internal safety warnings.
Silencing Internal Whistleblowers
Legal team ordered employee to delete posts raising concerns.
Failure to Pause Public Access
Management rejected recommendations to remove the tool from public use.
Regulatory Environment
Lack of Regulatory Standards
Absence of government mandates enforcing safety standards on AI tools.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
