This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Facebook's 'Year in Review' algorithm, designed to highlight positive memories from the past year, inadvertently caused emotional distress for many users by surfacing traumatic images, such as deceased loved ones or pets. The feature lacked an opt-out mechanism and failed to account for negative life events, leading to widespread criticism. Facebook subsequently apologized and implemented design changes to the feature.
Facebook’s “Year in Review” algorithm which compiled content in users’ past year as highlights inadvertently showed painful and unwanted memories to users, including death of family member.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
The algorithm failed to perform effectively under varying real-world conditions by failing to distinguish between positive and negative engagement, leading to the insensitive display of traumatic images.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The emotional distress was caused by the deployed Facebook Year in Review algorithm selecting and displaying sensitive images without user consent, which was an unintended outcome of the feature.
Added categories peak at 2: Minor (Psychological Harm, Direct)
Facebook's automated 'Year in Review' algorithm caused psychological distress by showing traumatic images to users. As an unintentional consumer product failure, it has negligible implications for national security.
The algorithm operated independently to select, format, and present user photos under the 'Year in Review' banner without explicit human intervention or planning for each individual user profile.
Minimal or No Risk: The system is a social media feature used for entertainment and nostalgia, which does not fall under the high-risk or prohibited categories of the EU AI Act.
The AI system's primary goal was to automatically curate and select highly-engaged photos and posts from a user's profile to generate a personalized 'Year in Review' summary.
The reports provide clear, consistent, and detailed accounts of the incident, including first-hand blog posts from the primary affected individual (Eric Meyer), statements from Facebook product managers, and multiple supporting examples from other users.
None
Technology
Engagement-biased Algorithm
Prioritized photos with high likes, which often include tragic events.
Lack of Text Semantic Analysis
Failed to filter out posts containing words like 'killed' or 'sad day'.
Absence of Easy Opt-Out
No obvious or simple way for users to dismiss or stop the feature.
Data Inputs
Unlabeled Emotional Valence
Engagement metrics failed to distinguish between grief and joy.
Human Factors
Design for Ideal User Only
Designers assumed a happy user, ignoring those experiencing crisis.
Process and Methods
Lack of Worst-case Scenario Testing
Product design failed to consider failure modes or edge cases.
Automatic Pre-filling of Content
Images were shown to users before asking for consent or a preview.
Management
Lack of Empathetic Product Design
Management prioritized engagement features over user emotional safety.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
