This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Facebook's 'On This Day' and 'Year in Review' algorithms automatically surfaced past posts to users to encourage nostalgia. However, these features frequently failed to account for personal trauma, displaying painful reminders of deceased loved ones, pets, and tragic events. Users reported significant emotional distress, highlighting the inability of the algorithms to discern the sentiment or appropriateness of the memories being resurfaced.
Facebook’s “On This Day” algorithm which highlighted past posts on a user’s private page or News Feed confronted unwanted and painful personal memories to its users.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
The algorithms failed to perform reliably under varying emotional contexts, exposing users to distressing content due to a lack of capability in sentiment analysis and contextual understanding.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The emotional distress was caused by the automated decisions of Facebook's curation algorithms, which produced unexpected negative outcomes while pursuing the goal of user engagement.
Added categories peak at 2: Minor (Psychological Harm, Direct)
Facebook's 'On This Day' and 'Year in Review' algorithms unintentionally resurfaced painful memories, causing emotional distress to users. While highlighting the limitations of sentiment analysis in consumer AI, the incident has negligible national security implications across all assessed categories.
The algorithms operated independently to select, format, and display historical posts in users' feeds based on learned patterns of engagement and dates, without requiring real-time human prompts or oversight for each output.
Minimal or No Risk: The system is a social media content curation and recommendation algorithm used for entertainment and engagement, which generally poses low systemic risk under the EU AI Act.
The primary function of the 'On This Day' and 'Year in Review' features is to automatically select, format, and present historical user content to encourage engagement and nostalgia.
The report provides clear, first-hand accounts of users affected by the algorithm's failures, along with expert commentary on the technical limitations of sentiment analysis. The causal link between the AI's curation decisions and the resulting psychological distress is explicitly documented.
None
Technology
Inability to Discern Sentiment
Algorithms cannot read emotional sentiment or context of posts.
Engagement-Driven Sorting
Algorithms surface highly engaged posts, which are often tragic.
Inflexible Filtering Rules
Filters only capture limited, explicitly marked life changes.
Data Inputs
Lack of Machine-Readable Loss Data
Users must document loss publicly for automated filters to work.
Unlabeled Tragic Content
Posts about tragedies lack explicit metadata indicating sadness.
Lack of Keyword Filtering
No automatic keyword filters for words like 'RIP' were active.
Human Factors
Unpredictable Emotional Response
Nostalgia is highly subjective and can trigger painful memories.
Public Persona Discrepancy
Curated happy public profiles hide complex internal emotions.
Process and Methods
Opt-Out Design Default
Feature is on by default with no way to completely opt out.
No Sentiment Analysis in Design
Processes lacked sentiment verification before surfacing memories.
Management
Prioritizing Nostalgia Engagement
Management pushed nostalgia features to drive user engagement.
Inadequate Risk Mitigation
Features deployed without fully assessing emotional harm risks.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
