This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A social media user reported concerns regarding the advertisement of PicSo AI on Instagram, alleging that the tool is being used to generate inappropriate and potentially sexually explicit content featuring girls. The user expressed concern about the potential for such generative AI tools to be misused for criminal purposes and encouraged reporting the platform to the Department of Justice.
PicSo AI, which appears to be getting advertised by Meta over Instagram, is allegedly being used for generating inappropriate content with an emphasis on "girls." This raises concerns about the misuse of generative AI technologies for creating offensive and potentially sexually explicit material that could be used for nefarious and criminal purposes.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
The incident involves an AI image generator being advertised and used to create sexually explicit content, which constitutes exposure to toxic and inappropriate content.
AI system
Due to a decision or action made by an AI system
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk is caused by the PicSo AI system generating explicit content, which is an intentional capability of the deployed application.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
A social media user reported that the PicSo AI image generator is being advertised on Instagram and potentially used to create sexually explicit content depicting minors. While presenting serious ethical and law enforcement concerns regarding child sexual exploitation and abuse (CSAE), the incident has minor direct implications for national security.
PicSo AI operates as an image generation tool that performs specific, well-defined functions based on direct user inputs without independent planning or decision-making.
Limited Risk: The system is an AI image generator producing synthetic content, which falls under transparency obligations for AI-generated content.
The AI system is a generative AI tool designed to produce images based on user prompts.
The report is a single social media post expressing personal concern about an advertisement. While the core complaint about PicSo AI's marketing is clear, there is no independent verification of the app's outputs or legal status.
The application may be a standard anime-style image generator whose suggestive marketing is being interpreted as targeting minors.
Technical details of the AI model, verification of the actual images generated, and any official regulatory or legal investigations into PicSo AI.
Technology
Unrestricted Generative AI Model
AI model allows generation of highly explicit or suggestive content of girls.
Flawed Ad Targeting Algorithms
Algorithms served inappropriate AI app ads alongside mainstream user feeds.
Data Inputs
Unfiltered Training Data
Training data likely included sexualized images of young-looking individuals.
Human Factors
User Demand for Explicit Content
Users actively seek and generate suggestive content using AI tools.
Process and Methods
Weak Ad Moderation Processes
Meta failed to detect and block inappropriate AI app advertisements.
Inadequate AI Safety Guardrails
PicSo lacked robust guardrails to prevent generating suggestive content.
Management
Prioritizing Ad Revenue Over Safety
Ad platforms host risky ads to maximize revenue from AI developers.
Exploitative App Monetization
PicSo developers explicitly targeted suggestive themes for user acquisition.
Regulatory Environment
Unclear AI Pornography Laws
Lax or ambiguous legal frameworks regarding AI-generated suggestive content.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
