This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
In early 2018, Facebook overhauled its News Feed algorithm to prioritize 'meaningful social interactions' (MSI) by heavily weighting reshares and comments. Internal research subsequently revealed that this change incentivized the creation and spread of sensationalist, toxic, and misinformative content, as such material generated higher engagement. Despite warnings from staff that the platform was becoming an 'angrier place' and harming democratic discourse, leadership resisted broad fixes due to concerns about potential declines in user engagement metrics.
After the “News Feed” algorithm had been overhauled to boost engagement between friends and family in early 2018, its heavy weighting of re-shared content was alleged found by company researchers to have pushed content creators to reorient their posts towards outrage and sensationalism, causing a proliferation of misinformation, toxicity, and violent content.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Highly personalized AI-generated misinformation creating “filter bubbles” where individuals only see what matches their existing beliefs, undermining shared reality, weakening social cohesion and political processes.
Also classified as
The algorithm directly amplified and exposed users to a higher prevalence of insults, threats, and angry, divisive content.
The algorithm failed to perform safely under real-world conditions, as its optimization metrics were easily exploited and gamed by political actors and publishers.
The algorithm overhaul caused ecosystem-scale degradation of public discourse, forcing political parties and publishers to adopt inflammatory tactics and polluting the information ecosystem with toxic, divisive content.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The harms were caused by the ranking decisions of the News Feed algorithm, which produced unintended negative side effects (amplifying outrage) while pursuing the goal of increasing meaningful social interactions.
Added categories peak at 5: Catastrophic (Psychological Harm, Inferred)
Facebook's 2018 News Feed algorithm overhaul unintentionally prioritized sensationalist, toxic, and misinformative content to drive engagement. This systemic amplification polluted the global information ecosystem, altered political campaigning strategies in multiple democracies, and facilitated the organization of the January 6 Capitol riot, posing notable threats to democratic sovereignty and societal stability.
The News Feed algorithm operates independently to rank and customize content for billions of users in real-time based on learned patterns and complex optimization formulas, without direct human intervention for individual ranking decisions.
Risk Level 1: Unacceptable Risk. The report describes an AI system used for behavioral influence and optimization that led to significant societal harm, including political polarization, the spread of toxic content, and the facilitation of civil unrest.
The primary function of the Facebook News Feed algorithm is to rank, curate, and recommend personalized content to users to maximize engagement and social interactions.
The assessment is based on a highly detailed investigative report by The Wall Street Journal, which reviewed internal Facebook memos, data science studies, and executive communications. This provides direct, authoritative evidence of the algorithm's design, its unintended consequences, and the internal awareness of these issues. There is very little ambiguity regarding the system's role or the resulting harms.
Facebook executives argued that societal divisions were growing long before social media, suggesting the algorithm was not the primary cause of polarization but merely reflected existing societal trends.
Technology
Flawed MSI Ranking Formula
The algorithm heavily weighted comments and reshares, promoting outrage.
Deep Reshare Amplification
Algorithm boosted content shared across long chains without verification.
Downstream MSI Optimization
Optimizing for downstream sharing rewarded highly controversial posts.
Data Inputs
Engagement-Based Metrics
Using raw shares and comments as proxies for meaningful interaction.
Lack of Sensitive Content Signals
Algorithm lacked inputs to distinguish healthy debate from toxic outrage.
Absence of Civic Content Signals
Initial lack of specialized signals for sensitive categories like politics.
Human Factors
Outrage Bias in User Engagement
Users are naturally more likely to comment on and share divisive content.
Publisher Adaptation to Outrage
Publishers shifted to sensationalism to maintain distribution.
Political Party Exploitation
Political actors used negative campaigning to maximize platform reach.
Process and Methods
Inadequate Impact Testing
Facebook deployed major algorithm changes without assessing societal risks.
Delayed Mitigation Deployment
Integrity team fixes were delayed or restricted to preserve metrics.
Overreliance on Engagement Surveys
Surveys focused on business health over systemic platform toxicity.
Management
Prioritizing Growth Over Safety
Management resisted fixes that would cause material tradeoffs with MSI.
Fear of Declining Engagement
Leadership feared users would leave the app if engagement continued to fall.
Resistance to Integrity Proposals
Executives blocked broad deployment of deep reshare mitigation.
Regulatory Environment
Absence of Algorithmic Regulation
No regulatory framework governed the design of recommendation algorithms.
Lack of External Oversight
No independent auditing of platform engagement metrics and impacts.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
