This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Washington State's Lottery launched an AI-powered mobile site called 'Test Drive a Win' that allowed users to superimpose their headshots into vacation-themed images. A user reported that the system generated an image of her face on a topless body. Following the report, the lottery organization took the website offline out of an abundance of caution.
An AI-powered website by Washington State's Lottery is reported to have inadvertently produced a softcore pornographic image of a user, leading to the site’s immediate shutdown out of caution.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The AI system failed to perform robustly under deployment conditions, failing to adhere to the strict parameters set by developers to keep generated individuals fully clothed.
The AI system generated and exposed the user to inappropriate, sexually explicit content (a topless image of herself) which violates community norms regarding pornography.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by an unexpected and unintended output generated by the AI system after it was deployed to the public.
Added categories peak at 2: Minor (Psychological Harm, Direct)
An AI-powered promotional website for the Washington State Lottery unintentionally generated a single sexually explicit image of a user. The incident represents a local software failure with negligible national security implications, and the site was promptly taken offline.
The AI system functioned as a tool that performed a specific, well-defined task of taking a user-uploaded headshot and superimposing it onto a vacation-themed background based on direct input.
Risk Level 3: The system is an AI-powered image generator that creates synthetic images (AI-generated content), which falls under the Limited Risk category requiring transparency obligations.
The system's primary purpose was to generate vacation-themed images incorporating the user's face. It functioned by generating synthetic images based on user headshots.
The reports provide clear, consistent details about the incident, including direct quotes from the affected user and official statements from the Washington Lottery spokesperson confirming the event and subsequent website shutdown.
None, as the facts of the generation and the subsequent shutdown are undisputed.
Technology
Generative Model Unpredictability
AI models can generate unintended NSFW outputs despite safety parameters.
Inadequate Image Filtering
Built-in safety filters failed to detect and block the topless output.
Data Inputs
User Headshot Uploads
Uploading arbitrary user faces combined with complex prompts caused anomalies.
Unconstrained Prompt Inputs
Vacation themes like 'swim with sharks' triggered unexpected associations.
Human Factors
Overreliance on AI Safeguards
Developers assumed pre-set rules would perfectly prevent inappropriate images.
Process and Methods
Lack of Pre-deployment Testing
Testing did not uncover edge cases where safety parameters could be bypassed.
Ineffective Output Validation
No real-time automated check to verify if generated images met clothing rules.
Management
Inadequate Risk Assessment
Management failed to fully assess risks of public-facing generative AI tools.
Delayed Site Takedown
Site remained online for days after the initial incident report was received.
Regulatory Environment
Lack of AI Safety Standards
No official regulations or standards governing public-facing AI deployments.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
