This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Instagram's automated advertising system selected a user's past post—a screenshot of a violent rape threat she had received—to promote the platform to her friends on Facebook. The system likely prioritized the post due to high engagement metrics, failing to detect or filter the abusive content within the image. This incident highlights ongoing issues with automated content moderation and advertising algorithms at Facebook and Instagram.
An Instagram user’s image containing violent content was reportedly used as advertisement on Facebook allegedly via automated means.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The algorithm lacked the capability to analyze text within images and filter out violent or abusive content before using it for promotions.
The AI system exposed users (the victim's sister and friends) to highly toxic, abusive, and violent content (a rape and death threat) by automatically generating an advertisement from the post.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by Instagram's automated engagement and advertising algorithms selecting and promoting an abusive post post-deployment without intending to spread toxic content.
Added categories peak at 2: Minor (Psychological Harm, Direct)
An automated Instagram algorithm mistakenly promoted a post containing a violent rape threat to a user's friends due to high engagement metrics. This incident represents a corporate content moderation failure with negligible national security implications, as it did not involve state actors, critical infrastructure, or population-scale manipulation.
The algorithm automatically selected, formatted, and served the post as an advertisement to Facebook users based on engagement metrics without human intervention or oversight.
Minimal or No Risk: The system is an automated ad delivery and engagement algorithm, which generally poses low or negligible risk to society under the EU AI Act.
The primary function of the algorithm was to deliver automated promotional posts (ads) to Facebook users to drive engagement back to Instagram.
The reports provide clear, firsthand accounts from the affected journalist and official statements from Instagram/Facebook confirming the mechanics of the incident.
None
Technology
Engagement-driven Ad Algorithm
The algorithm promoted posts solely based on high user engagement metrics.
Lack of Image Text Filtering
Systems failed to analyze and filter violent text embedded within images.
Data Inputs
Misinterpreted Engagement Signals
Likes and comments on a threat post were misconstrued as positive engagement.
No OCR Metadata Integration
Text within the screenshot was not ingested as metadata for safety filtering.
Human Factors
System Over-reliance
Operators relied fully on automation to select promotional content.
Process and Methods
No Human-in-the-Loop Review
Promotional notifications were dispatched without manual safety checks.
Inadequate Ad Safety Policies
Processes lacked specific guardrails for automated engagement ads.
Management
Prioritization of Growth Over Safety
Management prioritized platform engagement over robust content filtering.
Inadequate Risk Assessment
The risks of automated content promotion were not properly evaluated.
Regulatory Environment
Lack of Algorithmic Oversight
No regulatory standards existed to govern automated ad-targeting safety.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
