This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Twitter's saliency-based image cropping algorithm was found to exhibit racial and gender biases, consistently favoring white and lighter-skinned individuals over Black individuals in photo previews. Further research identified additional harms, including the marginalization of elderly people and individuals with disabilities, as well as biases related to language and beauty filters. In response, Twitter conducted internal audits, open-sourced their analysis, launched an algorithmic bias bounty program, and eventually moved away from automated cropping in favor of user-controlled image display.
Twitter's photo cropping algorithm was revealed by researchers to favor white and women faces in photos containing multiple faces, prompting the company to stop its use on mobile platform.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
The automated cropping caused representational harm by denying users the ability to represent themselves as they wished on the platform.
The image-cropping algorithm performed with unequal accuracy and effectiveness across different demographic groups, systematically favoring lighter-skinned individuals over Black individuals.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The bias was an unexpected and unintended consequence of the deployed saliency algorithm's design and training data.
Twitter's automated image-cropping algorithm exhibited systemic racial, gender, and demographic biases, favoring lighter-skinned individuals. While the incident affected millions of users globally and highlights significant challenges in AI fairness and representation, it has negligible direct national security implications and was resolved through corporate audits and system updates.
The saliency algorithm operates as a tool that takes an image input, calculates saliency scores, and outputs a cropped version deterministically based on those scores, without maintaining an internal state or planning.
Minimal or No Risk: The image-cropping algorithm is used for media presentation on a social media platform, which falls under simple applications with low risk to safety, though it does raise ethical and bias concerns.
The primary function of the algorithm was to automatically crop images to focus on the most salient parts for timeline previews.
The reports are highly detailed, including Twitter's own published scientific study, statistical results of demographic parity tests, and outcomes of a public bug bounty program. The technical mechanism (saliency tracking) and the nature of the bias are clearly documented.
The bias could be interpreted purely as a data representation issue (contrast and lighting in training datasets) rather than systemic racial prejudice in the code itself.
Technology
Saliency-based Neural Network
The algorithm crops images based on predicted visual interest, creating bias.
Contrast and Feature Bias
Algorithm is drawn to high contrast of light skin against dark backgrounds.
Data Inputs
Biased Eye-Tracking Data
Model was trained on eye-tracking data that can encode human biases.
Beauty Filter Skewed Inputs
Beauty filters skew saliency scores in favor of lighter, younger faces.
Limited Evaluation Datasets
Evaluation datasets excluded nonbinary or mixed-race demographic groups.
Human Factors
Potential Male Gaze Bias
Eye-tracking from straight male participants may encode sexualized views.
Loss of User Agency
Automated cropping prevents users from choosing how to represent themselves.
Process and Methods
Inadequate Initial Testing
Initial tests used too few images to detect demographic and gender bias.
Lack of Ethical Review Standards
No standard fairness metrics were available during model experimentation.
Management
Undefined AI Authority
Unclear who had final authority to decide if AI is too unfair for use.
Prioritizing Automation
Management prioritized automated speed over user choice and representation.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
