This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Internal Facebook research revealed that its engagement-based feed algorithms disproportionately exposed users with low digital literacy to graphic violence, nudity, and scams. Because these users lacked the technical skills to hide or report content, the algorithm misinterpreted their passive scrolling as approval, creating a feedback loop of harmful content. The research identified that these impacts were most severe for vulnerable populations, including older adults, people of color, and those with lower socioeconomic status.
Facebook feed algorithms were known by internal research to have harmed people having low digital literacy by exposing them to disturbing content they did not know how to avoid or monitor.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The algorithm performed worse for low-digital-literacy users (who are disproportionately older, people of color, and lower-income), leading to unequal safety outcomes.
Vulnerable users who followed coupon and savings pages were targeted and inundated with financial scams recommended by the algorithm.
The primary reported harm is the systematic and repeated exposure of vulnerable users to toxic content, including graphic violence, nudity, and scams, due to algorithmic feedback loops.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The harm was caused by the post-deployment operations of Facebook's News Feed algorithm, which unintentionally created a toxic feedback loop by misinterpreting passive scrolling as user approval.
Added categories peak at 4: Severe (Psychological Harm, Inferred)
Internal Facebook research revealed that its News Feed recommendation algorithm created toxic feedback loops that disproportionately exposed low-digital-literacy users to graphic violence, scams, and political propaganda. While the societal impact is widespread, the direct national security implications are minor, primarily manifesting as passive amplification of foreign propaganda and erosion of democratic discourse.
The News Feed algorithm operates independently without explicit planning, responding to real-time user inputs (like scrolling speed and clicks) using learned patterns to make automated decisions about what content to display next.
Unacceptable Risk: The report describes an AI system that systematically exploits the vulnerabilities of users based on age, education, and socioeconomic status, leading to repeated exposure to disturbing and harmful content.
The primary function of the Facebook News Feed algorithm is to curate and recommend content to users based on their predicted preferences and engagement behaviors.
The assessment is supported by extensive internal research documents leaked by whistleblower Frances Haugen, including specific survey metrics (e.g., 11.4% more nudity, 13.4% more graphic violence) and demographic correlations. The reports from multiple reputable news organizations consistently describe the same algorithmic mechanisms and harms.
One could argue the primary failure is human digital illiteracy rather than the AI system itself, but the internal documents explicitly acknowledge the algorithm's role in creating a harmful feedback loop.
Technology
Algorithmic Feedback Loop
AI misinterprets passive scrolling as approval, feeding more bad content.
Inaccurate Content Classifier
AI fails to distinguish violent content from non-violent content.
Default Feed Nudity Exposure
Default feed includes nudity and borderline content unless filtered.
Data Inputs
Lack of Negative Feedback Data
Low-literacy users do not flag content, leaving AI without signals.
Pending Group Invites as Input
Pending group invites feed unapproved posts directly into user feeds.
Imperfect Misinformation Signals
Algorithms struggle to recognize misleading content accurately.
Human Factors
Low Digital Literacy
Users do not know how to use hide, unfollow, or block tools.
Ignoring Neutral Warning Prompts
Low-tech users do not understand or they ignore neutral warnings.
Sensory Overload
Newcomers face sensory overload and do not understand the algorithm.
Process and Methods
Inadequate Warning Language
Neutral warning prompts fail to arouse skepticism in users.
Lack of Easy Hide Deployment
The Easy Hide button was tested successfully but not deployed.
Aggregate Reporting Bias
Aggregate data reporting hides extreme negative experiences of subgroups.
Management
Prioritizing Neutral Language
CEO mandated neutral tone to avoid false positives, weakening alerts.
Growth Over Vulnerable Safety
Management failed to prioritize safety for low-literacy users.
Under-enforcement Preference
Management preferred to err on the side of under-enforcement.
Regulatory Environment
Lack of Algorithmic Regulation
Absence of regulations allowed harmful feed defaults to persist.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
