This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Reports indicate that various AI models, including Meta's LLaMA, Stability AI's Stable Diffusion, CharacterAI, and OpenAI's ChatGPT, are being used to generate graphic, violent, and illegal sexual content. The incident highlights the tension between open-source AI development and the potential for misuse, including the generation of child sexual abuse material.
Meta's open-source large language model, LLaMA, is allegedly being used to create graphic and explicit chatbots that indulge in violent and illegal sexual fantasies. The Washington Post highlighted the example of "Allie," a chatbot that participates in text-based role-playing allegedly involving violent scenarios like rape and abuse. The issue raises ethical questions about open-source AI models, their regulation, and the responsibility of developers and deployers in mitigating harmful usage.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The incident highlights widespread vulnerabilities where users easily circumvent and manipulate the NSFW guardrails of various AI systems.
The incident primarily involves the generation of and exposure to highly toxic, violent, and illegal content, including child sexual abuse material and violent abuse fantasies.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risks and harms are driven by human users intentionally bypassing guardrails and utilizing deployed AI models to generate toxic, violent, and illegal content.
Added categories peak at 4: Severe (Child Sexual Exploitation & Abuse (CSEA), Inferred)
The widespread circumvention of AI guardrails to generate child sexual abuse material (CSAM) and graphic, violent sexual content represents a substantial threat to societal stability and human rights. While it does not directly threaten critical physical infrastructure or state sovereignty, the systemic abuse of open-source models presents major long-term challenges for law enforcement and regulatory bodies globally.
The AI systems function as conversational assistants and image generation tools that respond to user prompts to generate text and images, requiring human input and oversight to function.
Limited Risk: The report describes chatbots and AI-generated content generators, which fall under Risk Level 3 as they require transparency obligations to ensure users are aware they are interacting with AI.
The report primarily focuses on users utilizing large language models to create interactive sexbots and generate sexual text, with a secondary focus on image generators creating CSAM.
The reports clearly detail the systems involved (LLaMA, Stable Diffusion, ChatGPT, Character.AI) and the specific nature of the misuse (NSFW content, violent fantasies, CSAM). The boundary rules clearly point to 1.2 (Exposure to toxic content) and CSEA.
One could argue the primary domain is 4.3 due to the creation of sexual imagery, but boundary rule B1 clarifies that non-targeted sexualized content falls under 1.2.
Technology
Open-source LLM Availability
Open-source release of LLaMA allows users to bypass hosted guardrails.
Imperfect Algorithmic Guardrails
Corporate guardrails are easily bypassed with clever prompting.
Human Factors
User Circumvention of Rules
Communities share tips on Reddit to bypass NSFW filters.
Malicious Intent by Creators
Creators intentionally build bots for violent and illegal fantasies.
Process and Methods
Lack of Model Gatekeeping
Releasing model weights directly makes post-deployment control impossible.
YouTube Tutorials on Fine-tuning
Developers share video guides on how to build custom bots using LLaMA.
Management
Prioritization of Innovation
Meta chose open-sourcing to advance technology over safety controls.
Regulatory Environment
Unregulated AI Field
Lack of government regulations governing AI safety and model releases.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
