This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A lawsuit alleged that Snapchat's 'Quick Add' recommendation algorithm facilitated sexual abuse by connecting a minor with convicted sex offenders on multiple occasions. The court dismissed the complaint based on Section 230 immunity, ruling that the platform's design of recommendation systems is a protected activity. The case highlights ongoing concerns regarding the role of algorithmic design in child safety and the limitations of current legal frameworks in addressing these harms.
A judge ruled Snapchat not liable under Section 230 after its algorithm connected a minor with convicted sex offenders on multiple occasions, leading to sexual assaults first in 2019 and again in 2021. The platform's "Quick Add" feature was implicated in facilitating the connections between the minor and the offenders.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The algorithmic recommendation directly led to the minor being exposed to inappropriate images, threats, and sextortion on the platform.
The Snapchat Quick Add algorithm failed to perform safely by recommending convicted sex offenders to a minor, demonstrating a critical lack of safety capabilities and robustness in its matching logic.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The harm was driven by the post-deployment operation of Snapchat's Quick Add recommendation algorithm, which unintentionally matched a minor with sexual predators.
Added categories peak at 4: Severe (Child Sexual Exploitation & Abuse (CSEA), Inferred)
A domestic child safety incident where Snapchat's Quick Add recommendation algorithm matched a minor with convicted sex offenders, leading to abuse. While representing a severe failure in algorithmic safety and child protection, the national security implications are minor and manageable within standard domestic legal and law enforcement frameworks.
The Quick Add feature operates as an autonomous recommendation system, independently generating and presenting friend suggestions to users based on underlying algorithmic patterns without direct human intervention for each match.
High Risk: The system's algorithmic recommendations have significant implications for safety, fundamental rights, and public interests, specifically the physical safety and protection of minors from sexual predators.
The primary function of the Quick Add algorithm is matchmaking by recommending user profiles to other users to facilitate social connections.
The report provides clear, factual details of the legal case, the specific occurrences of harm to C.O., and the role of Snapchat's Quick Add recommendation algorithm. The legal outcomes and arguments from both sides are well-documented.
The defense argued that the harm was entirely caused by third-party actors and that Section 230 shields the platform, framing it as a third-party content issue rather than an algorithmic design failure.
Technology
Quick Add Algorithm Design
Recommendation system directed minor users to connect with sex offenders.
Failed Threat Detection
System failed to detect sexually explicit photos exchanged on the platform.
Data Inputs
Lack of Profile Verification
Sex offenders easily registered and maintained active accounts on the app.
Ignored Harassment Reports
System did not utilize user reports of harassment to block predator accounts.
Human Factors
Vulnerability of Minor Users
Young users were easily manipulated and pressured by adult predators.
Process and Methods
Unsafe Recommendation Logic
Algorithm connected predators with young girls based on engagement metrics.
Inadequate Account Removal
Reported predator accounts remained active for years without being banned.
Management
Neglect of Minor Safety
Management failed to implement safeguards despite knowing predators used the app.
Failure to Mitigate Known Risks
Prioritized engagement features over protecting vulnerable minor users.
Regulatory Environment
Section 230 Immunity Shield
Broad legal immunity protected the platform from liability over algorithm design.
Lack of Safety Regulations
Absence of legislative standards allowed unsafe design features to persist.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
