This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Meta's AI-generated sticker tool, powered by Llama 2, was found to produce inappropriate and offensive content, including sexualized imagery and depictions of public figures in compromising situations. Users discovered that the tool's safety filters could be bypassed using typos or specific prompts, leading to the generation of problematic stickers. This incident highlights challenges in content moderation for generative AI features deployed in consumer messaging platforms.
Facebook Messenger AI stickers, a feature by Meta, allows users to generate personalized stickers via AI for use in conversations. While the feature has been praised for its creativity, it has also stirred controversy for its alleged production of inappropriate or offensive content. This has raised questions about the effectiveness of Meta's content moderation measures and the ethical responsibilities associated with AI-driven content generation.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The system's safety filters were easily bypassed using typos or descriptive workarounds, showing a lack of robustness in content moderation.
The AI tool generated and exposed users to highly inappropriate, offensive, and sexualized content, including child soldiers and nude illustrations of public figures.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The generation of inappropriate and offensive stickers was an unexpected and unintentional outcome of deploying the Llama 2-powered sticker tool.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
Meta's Llama 2-powered sticker tool generated highly inappropriate and offensive stickers, including depictions of child soldiers and nude illustrations of Canadian Prime Minister Justin Trudeau, due to easily bypassed safety filters. The national security impact remains minor, primarily highlighting the ongoing challenges of content moderation and information security in consumer-facing generative AI tools.
The AI system operates as a Tool, performing the specific, well-defined function of generating chat stickers based on direct text prompts from users without maintaining an ongoing autonomous state or planning.
Limited Risk: The system is an AI-generated content tool used to create synthetic images (stickers), which falls under the category of AI-generated content requiring transparency obligations.
The primary function of the Llama 2-powered tool is to generate unique stickers (images) from text prompts for use in messaging apps.
The report clearly details the AI system involved (Meta's Llama 2-powered sticker tool), the platform (Facebook Messenger), the nature of the generated content, and how the safety filters were bypassed. There is little ambiguity regarding the events.
None. The incident is a straightforward case of generative AI safety filter failure leading to inappropriate content generation.
Technology
Weak Algorithmic Safeguards
The model generated inappropriate images like nude figures and weapons.
Easily Bypassed Filters
Users bypassed blocked words using simple typos or descriptive text.
Data Inputs
Unfiltered Prompt Inputs
Certain prompts like 'World Trade Center' bypassed filters without warnings.
Human Factors
Adversarial User Testing
Users intentionally entered prompts to test limits and generate bad images.
Process and Methods
Insufficient Pre-Release Testing
Live user testing was used to find obvious vulnerabilities in the tool.
Management
Risky Feature Rollout
Meta deployed the tool to select users despite unresolved safety issues.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
