This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
In June 2015, Google's Photos app automatically misclassified photos of a Black man and his friend as 'gorillas'. The incident, which was publicized by the affected user on Twitter, drew widespread criticism for the AI's racist output. Google apologized and initially attempted to fix the algorithm, but eventually removed the 'gorilla' label and search functionality entirely to prevent further harm.
Google Photos image processing software mistakenly labelled a black couple as "gorillas."
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
The system's output associated Black individuals with a deeply offensive racist trope, representing them as primates.
The AI system exhibited lower accuracy and performance for darker-skinned individuals due to a lack of diverse training data, leading to offensive misclassifications.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The misclassification was an unexpected error made by the Google Photos image recognition AI after it was deployed to the public.
Added categories peak at 2: Minor (Psychological Harm, Direct)
An unintentional post-deployment misclassification by Google Photos in 2015 labeled Black individuals as 'gorillas'. While causing significant public outcry and highlighting critical issues of racial bias in AI training datasets, the incident has negligible national security implications, representing a minor societal impact managed through corporate software adjustments.
The system acts as an Assistant by automatically tagging and organizing photos to help users manage their collections, but it relies on human oversight to correct mislabeling errors.
Minimal or No Risk: The system is a consumer photo organization and search application, which poses low or negligible risk to safety or fundamental rights under the EU AI Act.
The primary function of the Google Photos AI was to automatically tag images with descriptive labels to facilitate search and organization.
The reports provide consistent, detailed accounts of the 2015 Google Photos incident, including direct quotes from the affected user, Google executives, and subsequent technical investigations by Wired. The facts of the misclassification and Google's subsequent workaround are well-documented across multiple reputable sources.
None
Technology
Nascent Computer Vision Models
Algorithms rely on basic patterns and lack human common sense.
Black Box Nature of ML
Creators struggle to understand how models make classification decisions.
Inadequate Contrast Processing
Image sensors and software struggle with dark skin tones and low lighting.
Data Inputs
Lack of Diverse Training Data
Training datasets lacked sufficient images of dark-skinned individuals.
Biased Dataset Selection
Bias in collected images led to poor representation of people of color.
Human Factors
Lack of Diversity in Tech Teams
Homogeneous engineering teams failed to anticipate or test for diverse users.
Overreliance on Automation
Assuming automated tagging would perform flawlessly without human oversight.
Process and Methods
Insufficient QA Testing
The app was not tested on a sufficiently diverse focus group before release.
Ineffective Workaround Fixes
Blocking primate labels rather than retraining the underlying neural network.
Lack of Pre-release Employee Testing
Not enough employees were asked to test the feature before its public debut.
Management
Prioritization of Speed to Market
Releasing cutting-edge tech quickly with the plan to fix bugs post-launch.
Lack of Diversity Initiatives
Slow progress in recruiting and retaining Black and Latino technical staff.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
