This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
FaceApp, a popular selfie-editing application, faced significant backlash on two occasions in 2017. First, users discovered that its 'hot' filter consistently lightened the skin of people of color, which the company attributed to training set bias. Later, the app introduced 'ethnicity' filters that allowed users to change their appearance to look like different races, which was widely condemned as digital blackface and yellowface. The company eventually removed both features following public outcry.
FaceApp is criticized for offering racist filters.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.
Also classified as
The neural network exhibited a lack of robustness due to training set bias, failing to perform equitably across different demographic groups.
The AI system misrepresented users of color by automatically whitening their skin to make them look 'hot' and generated offensive, stereotyped representations of different races through its ethnicity filters.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was primarily driven by the biased outputs of FaceApp's neural network and the automated generation of racially stereotyped filters.
Added categories peak at 2: Minor (Psychological Harm, Direct)
FaceApp's release of racially biased and offensive filters sparked global public backlash, highlighting issues of algorithmic discrimination and consumer data privacy. While the incident caused notable societal offense and exposed millions of users to biased AI outputs, its direct threat to national security remains negligible to minor.
FaceApp functions as an image-processing tool where users upload a photo and select a specific filter to apply a deterministic transformation to that image.
Minimal or No Risk: The system is an entertainment application used for simple photo filtering, which poses low or negligible risk to society under the EU AI Act guidelines.
The primary function of FaceApp is to retouch and modify user selfies using neural networks to change facial characteristics.
The reports are highly consistent across multiple reputable news outlets, providing clear details on the app's functionality, the nature of the public backlash, and statements from the company's CEO.
None
Technology
Biased Neural Network
The deep learning model associated whiter skin with hotness.
Stereotyping Algorithms
Filters altered features like nose and lips based on racial stereotypes.
Data Inputs
Homogeneous Training Data
Training set was biased, containing mostly photos of white people.
Lack of Diverse Datasets
The AI lacked representative images for non-white ethnicities.
Human Factors
Cultural Insensitivity
Developers lacked awareness of blackface and racial parody history.
Algorithmic Overreliance
Assuming neural net outputs are objective and equal in all aspects.
Process and Methods
Inadequate Pre-release Testing
The app was launched without testing for racial bias or offense.
Poor Risk Assessment
No process to evaluate the social impact of ethnicity-altering AI.
Management
Prioritizing Novelty Over Safety
Management focused on viral growth and photorealism over ethics.
Defensive Initial Response
CEO initially defended the filters as equal instead of removing them.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
