This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Amazon developed an experimental AI-powered recruiting tool between 2014 and 2017 intended to automate the screening of job applicants. The system, trained on a decade of historical hiring data that was predominantly male, learned to exhibit gender bias by penalizing resumes that included the word 'women's' and downgrading graduates from all-women's colleges. The project was ultimately abandoned after the company determined the bias could not be reliably eliminated.
Between 2014 and 2017, Amazon reportedly developed an AI-powered recruiting tool to score job applicants, trained on a decade of resumes purportedly drawn largely from men. Media reports say the system learned to favor male candidates, penalizing terms like "women's" and graduates from certain all-women's colleges. Efforts to remove these biases reportedly did not guarantee fairness, and the project was ultimately abandoned. Amazon reportedly states recruiters never solely relied on the tool.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.
Also classified as
The system suffered from a lack of capability and robustness, frequently recommending unqualified candidates at random due to data issues.
The AI recruiting engine systematically downgraded female applicants by penalizing resumes containing gender-specific terms and women's colleges, leading to unfair discrimination.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Pre-deployment
Occurring before the AI is deployed
The gender bias was an unexpected outcome of the AI system learning from historical data during its experimental trial phase before full deployment.
Amazon developed an experimental AI recruiting tool that exhibited gender bias by penalizing resumes with female-associated terms. The project was abandoned in 2017 during its trial phase without being fully deployed. The incident has negligible national security implications, representing a private sector algorithmic bias issue with no impact on critical infrastructure, sovereignty, or intelligence.
The AI system was designed to perform a specific, well-defined function of reviewing resumes and assigning a score from one to five stars based on direct inputs. It operated as a screening tool to assist recruiters rather than acting as an autonomous agent with its own goals.
High Risk: The system is an employment and recruitment tool used to screen job candidates. Under the EU AI Act, 'Employment, workers management and access to self-employment' systems, such as those used for recruiting or selecting natural persons, are classified as High Risk.
The system's primary function was to evaluate job applications and resumes to identify and rank the top candidates for technical roles.
The reports provide consistent, detailed accounts of Amazon's experimental recruiting tool, its development timeline, the specific mechanisms of its gender bias, and the decision to scrap the project. Multiple independent sources and Amazon's own statements confirm the core facts of the incident.
None
Technology
Algorithmic Pattern Matching
The model learned to associate male-dominated terms with success.
Inability to Guarantee Neutrality
Editing specific terms did not stop the model from finding new biases.
Poor Recommendation Logic
Glitches in model logic led to recommending unqualified candidates.
Data Inputs
Male-Dominated Training Data
Ten years of past resumes were predominantly from male applicants.
Biased Feature Representation
The model penalized 'women's' and favored verbs like 'executed'.
Unstructured CV Data Glitches
Underlying data issues caused random and unqualified recommendations.
Human Factors
Unconscious Bias in Resume Language
Societal biases led to gendered language preferences in resumes.
Overreliance on Historical Patterns
Engineers assumed past hiring patterns represented ideal candidates.
Lack of Diversity in Tech Teams
Homogeneous development teams failed to anticipate gender bias issues.
Process and Methods
Flawed Model Optimization Goal
Optimizing for past hiring patterns scaled historical biases.
Inadequate Pre-deployment Testing
Bias issues were discovered only after the model was in trial phases.
Flawed Performance Metrics
The 1-to-5 star rating system oversimplified complex human attributes.
Management
Prioritizing Automation Over Equity
Management sought a 'holy grail' to fully mechanize the hiring process.
Devaluation of HR Expertise
Attempted to replace empathetic, human-centric HR skills with software.
Inadequate Initial Risk Assessment
Management failed to assess the risk of reinforcing historical biases.
Regulatory Environment
Lack of Regulatory Standards
No legal frameworks existed to enforce algorithmic fairness in hiring.
Absence of Mandatory Disclosure
No laws forced companies to disclose the use of automated hiring tools.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
