This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Transgender Uber drivers were repeatedly deactivated from the platform because the company's 'Real-Time ID Check' facial recognition system, powered by Microsoft Cognitive Services, failed to recognize them as their appearance changed during gender transition. This automated process resulted in loss of income for drivers who were forced to travel to in-person support centers to resolve the errors. The incident highlights issues with algorithmic bias and lack of inclusivity in automated identity verification systems.
Transgender Uber drivers reported being automatically deactivated from the app due to Real-Time ID Check failing to account for difference in appearance of people undergoing gender transitions.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
The automated deactivations led to loss of income and increased barriers to employment for transgender drivers.
The facial recognition system performed with lower accuracy for transgender drivers undergoing transition, leading to automated deactivations and unequal access to the platform.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The deactivations were caused by the automated Real-Time ID Check AI system failing to recognize transgender drivers post-deployment, which was an unintentional outcome of its security goals.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
An automated identity verification system used by Uber and powered by Microsoft failed to recognize transgender drivers undergoing gender transition, leading to erroneous account deactivations. This incident highlights ongoing civil rights and algorithmic bias concerns within commercial AI systems, but carries negligible national security implications.
The Real-Time ID Check system automatically prompts drivers, compares their selfies to photos on file using Microsoft Cognitive Services, and autonomously suspends accounts if a match is not found, operating without real-time human intervention.
High Risk: The system is used for biometric identification (facial recognition) and as part of an employment/worker management platform, both of which are classified as High Risk under the EU AI Act.
The primary function of the Real-Time ID Check is to verify the identity of the driver using facial recognition technology. It also serves as a workforce monitoring tool to ensure authorized individuals are operating the accounts.
The report clearly details the AI system used (Microsoft Cognitive Services / Uber Real-Time ID Check), the specific failure mode (failing to match transitioning transgender drivers), and the resulting financial and operational harms. The accounts of the two drivers are consistent and supported by Uber's own statements.
None. The causal link between the facial recognition failure and the driver deactivations is clear and acknowledged by Uber.
Technology
Inflexible Facial Recognition
Microsoft Cognitive Services failed to recognize transitioning faces.
Algorithmic Red Flags
System flagged natural facial changes as potential fraud.
Data Inputs
Static Reference Photos
System compared current selfies against outdated driver's licenses.
Lack of Photo History Integration
Uploaded transition photos were not used to update the matching baseline.
Human Factors
Physical Appearance Changes
Drivers undergoing gender transition naturally changed their appearance.
Process and Methods
Overly Automated Deactivation
System automatically suspended accounts without manual review.
Inconvenient Resolution Process
Drivers had to travel to physical support centers to resolve issues.
Management
Prioritization of Automation
Uber prioritized automated efficiency over edge-case driver safety.
Inadequate Risk Assessment
Management failed to assess risks of facial recognition for trans drivers.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
