This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Lensa AI, a popular photo-editing app, faced significant controversy after users reported that its 'Magic Avatars' feature generated sexually explicit and nude images from modest selfies. The app was found to disproportionately sexualize women and Asian women, while also exhibiting racial biases such as skin lightening. Additionally, the app raised ethical concerns regarding the use of artists' work in its training data and the potential for generating non-consensual sexual content, including imagery of minors.
Lensa AI's "Magic Avatars" were reportedly generating sexually explicit and sexualized features disproportionately for women and Asian women despite not submitting any sexual content.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The app systematically altered users' features based on gender and race, such as lightening skin, anglicizing features, and applying sexualized stereotypes to women.
The underlying Stable Diffusion model scraped and reproduced copyrighted artistic styles without consent, devaluing the work of digital artists.
The Lensa AI system directly generated and exposed users to highly inappropriate, sexually explicit, and toxic content, including unsolicited nudes of themselves and sexualized images of children.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The generation of sexually explicit, biased, and child-sexualized images was an unexpected and unintended outcome of deploying the Lensa AI avatar feature.
Added categories peak at 3: Substantial (Psychological Harm, Inferred)
Lensa AI's Magic Avatars feature demonstrated significant gender and racial biases, alongside generating non-consensual explicit imagery and child-related sexualized content. While representing notable societal and privacy concerns, the incident has negligible direct impact on national security, falling primarily under consumer safety and regulatory domains.
Lensa AI operates as a tool that performs a specific, well-defined function of generating stylized avatars based on user-submitted photos. It operates on direct input without maintaining an ongoing interactive state or planning capabilities.
Limited Risk: The app generates synthetic images and deepfakes of users, which under the EU AI Act are subject to transparency obligations to ensure users are aware the content is AI-generated.
The primary function of Lensa AI is to generate computer-generated images and artistic avatars of users based on their uploaded selfies.
The reports provide consistent, detailed, and first-hand accounts of Lensa AI's behavior, including specific tests run by journalists that confirmed the generation of explicit content and biased outputs. The technical mechanism is well-documented, and Prisma Labs' responses are included.
None. The evidence of bias and unwanted sexualization is clear and replicated across multiple independent reports.
Technology
Unfiltered Stable Diffusion Model
Open-source model lacks technical constraints to block unsafe generations.
Biased CLIP Neural Network
CLIP associates women with sexual content and men with professional roles.
Disabled or Bypassed NSFW Filter
The app lacked active filters, allowing generation of explicit images.
Data Inputs
Uncurated LAION-5B Dataset
Dataset contains billions of internet images with deep societal biases.
Non-Consensual Artist Data
Training data used copyrighted art without permission or licensing.
Lack of Input Photo Verification
System does not verify if uploaded photos belong to users or minors.
Human Factors
User Violations of Terms
Users uploaded photos of minors and photoshopped images to bypass rules.
Societal Biases in Internet Data
Human prejudices online were captured in data and mirrored by the AI.
Intentional Harassment Motives
Bad actors exploit the tool to generate non-consensual revenge porn.
Process and Methods
Inadequate Content Moderation
Lack of active output filtering allowed explicit images to reach users.
Flawed Gender Classification
Binary gender selection triggers pre-defined sexualized artistic styles.
Absence of Artist Opt-Out Process
No mechanism was provided for artists to remove their work from training.
Management
Prioritization of Rapid Growth
App launched without safety layers to capture market demand quickly.
Failure to Enforce Terms
Prisma Labs did not actively block uploads of minors' or nude photos.
Inadequate Risk Assessment
Management failed to foresee generation of deepfakes and child imagery.
Regulatory Environment
Lack of AI Content Regulation
No legal frameworks govern the generation of AI-synthesized pornography.
Unregulated Data Scraping
Weak copyright laws allowed scraping of artists' work without consent.
No Age Verification Mandates
Absence of strict legal requirements to verify user age before generation.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
