This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Major tech companies including Google and Apple have disabled or restricted primate identification in their photo recognition software to avoid repeating past incidents where Black individuals were incorrectly labeled as primates. This approach of 'walling off' features rather than fixing underlying data bias has been criticized by experts as a failure to address the root cause of algorithmic discrimination. The incident highlights the ongoing challenge of ensuring AI systems are trained on diverse datasets to prevent harmful racial stereotyping.
Eight years after Google Photos mislabeled images of Black individuals as "gorillas," image recognition software by Google, Apple, Amazon, and Microsoft still shows signs of either avoiding or inaccurately categorizing primates. Tests reveal that Google and Apple Photos refrain from labeling primates altogether, possibly to avoid the risk of perpetuating racial stereotypes. Microsoft OneDrive fails to identify any animals, while Amazon Photos overgeneralizes in its labeling.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
The system's output mislabeled Black individuals as primates, perpetuating harmful and offensive racial stereotypes.
The AI system performed with lower accuracy for Black individuals due to biased training data, leading to unequal outcomes and offensive misclassifications.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The racist mislabeling was an unexpected and unintentional outcome of the AI system's image classification process after it was deployed to the public.
Added categories peak at 2: Minor (Psychological Harm, Direct)
The incident involves commercial photo-recognition software misclassifying Black individuals due to biased training data. While highlighting significant ethical concerns and algorithmic limitations in consumer technology, it poses negligible threat to national security, critical infrastructure, sovereignty, or strategic economic interests.
The AI systems in Google Photos and Apple Photos function as image classification and search tools, performing specific, well-defined labeling tasks based on direct user inputs without independent planning.
Minimal or No Risk: The consumer photo-tagging and search applications described in the report pose minimal risk to safety or fundamental rights under the EU AI Act, falling under Risk Level 4.
The AI system's primary function is to analyze and classify images to automatically apply labels to people, places, and things, enabling users to search their photo collections.
The report provides clear, historical, and verified details about the Google Photos incident, including statements from former employees, spokespeople, and independent testing of competitor apps.
None.
Technology
Inability to Distinguish Dark Skin
Models confused darker-skinned humans with primates due to representation gaps.
Walling Off Malfunctioning Features
Disabling primate labels entirely instead of fixing algorithmic flaws.
Data Inputs
Lack of Diverse Training Data
Insufficient photos of Black people in the training dataset caused bias.
Troubling Data Collection Tactics
Targeting vulnerable groups to quickly gather dark-skinned facial scans.
Human Factors
Overreliance on AI Technology
Society and developers put too much trust in unproven computer vision systems.
Process and Methods
Inadequate Pre-release Testing
Not asking enough diverse employees to test the feature before public debut.
Ineffective Model Verification
Failing to detect systemic classification errors during internal validation.
Management
Prioritizing Speed Over Safety
Rushing public release without thoroughly checking for demographic biases.
Risk Avoidance Over Resolution
Deciding to disable primate recognition to mitigate reputational risk.
Regulatory Environment
Absence of AI Bias Regulations
No regulatory standards forcing companies to audit training data for diversity.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
