This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Private medical record photos were discovered in the LAION-5B dataset, which is used to train popular generative AI models like Stable Diffusion and Imagen. The affected individual, an AI artist, discovered her photos had been scraped without consent after her doctor's death. This incident highlights significant privacy concerns regarding the inclusion of sensitive, non-consensual data in large-scale training sets for commercial AI products.
Text-to-image models trained on the LAION-5B dataset, such as Stable Diffusion and Imagen, have been reported to reproduce or approximate images resembling private medical record photos allegedly included in the training data without documented consent or established mechanisms for removal.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
The incident involves the unauthorized scraping and inclusion of private, highly sensitive medical photos of an individual's face into a public AI training dataset.
Human
Due to a decision or action made by humans
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The privacy violation was caused by human decisions to scrape and compile web images into the LAION-5B dataset without verifying consent, which was an unintentional outcome of pursuing a large-scale open dataset.
Added categories peak at 2: Minor (Psychological Harm, Direct)
The inclusion of private medical photos in the LAION-5B training dataset highlights significant data privacy and consent challenges in the AI supply chain. While distressing for affected individuals and indicative of systemic data curation issues, the incident does not pose a direct threat to national security, critical infrastructure, or sovereignty.
The AI models (Stable Diffusion, Imagen) and the LAION-5B dataset function as tools for image generation and training, operating on direct inputs to produce specific outputs without independent planning or agency.
Limited Risk: The generative AI models like Stable Diffusion and Imagen are used to generate synthetic images, which under the EU AI Act are subject to transparency obligations to inform users that the content is AI-generated.
The primary function of the AI models trained on the LAION-5B dataset, such as Stable Diffusion and Imagen, is image generation from text descriptions.
The report is highly detailed, featuring direct interviews with the affected individual, confirmation of the images' presence in the dataset by the journalists, and responses from the CEO of Stability AI and a LAION engineer.
None
Technology
Indiscriminate Web Scraping
LAION automated tools scraped web URLs without verifying image consent.
Lack of Untraining Capability
AI models cannot easily un-learn or remove specific trained images.
Data Inputs
Leaked Private Medical Photos
Private medical photos were stolen or leaked online after doctor's death.
Lack of Consent Metadata
Scraped image URLs lacked metadata indicating consent status or privacy.
Human Factors
Unauthorized Upload by Third Party
An unknown actor uploaded private medical files to the public internet.
Process and Methods
No Simple Opt-Out Process
No mechanism exists to remove images from dataset without sharing more data.
Diffusion of Responsibility
Dataset creators point to hosting sites rather than taking responsibility.
Management
Prioritizing Scale Over Consent
LAION prioritized dataset size and openness over individual privacy rights.
Lack of Data Governance
No proactive vetting of dataset contents for sensitive medical imagery.
Regulatory Environment
Permissive Scraping Laws
US laws permit scraping publicly accessible web data without explicit consent.
Inadequate GDPR Alternatives
Privacy protections like GDPR do not easily cover non-EU citizens.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
