This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The social networking app Giggle utilized Kairos facial recognition software to perform gender verification on new users. The system faced significant criticism for discriminatory performance, specifically failing to accurately classify women of color, and was used by the platform's founder to exclude transgender women. The app also faced scrutiny for its extensive data collection practices and privacy policies.
A social networking platform, Giggle, allegedly collected, shared to third-parties, and used sensitive information and biometric data to verify whether a person is a woman via facial recognition, which critics claimed to be discriminatory against women of color and harmful towards trans women.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
The platform's onboarding process was intentionally designed and used to systematically exclude transgender women.
The app's privacy policy allowed the collection and sharing of highly sensitive personal data including sexual practices and health information.
The facial recognition software exhibited unequal performance across demographic groups, specifically misclassifying darker-skinned women at a much higher rate of 22.5% and failing to verify transgender women.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident involves the post-deployment failure of the Kairos facial recognition AI to accurately classify users' gender, resulting in unintentional discriminatory exclusions.
Added categories peak at 2: Minor (Psychological Harm, Direct)
The Giggle social media app utilized biased facial recognition software for gender verification, resulting in discriminatory exclusion of trans women and women of color, coupled with highly invasive data collection practices. While raising significant ethical and privacy concerns, the incident remains a localized commercial issue with negligible direct national security implications.
The Kairos software acts as a tool that takes a single selfie as input and outputs a gender classification percentage. It does not exhibit autonomous planning or maintain an internal state beyond this deterministic function.
High Risk: The system uses biometric identification and categorization of natural persons, which has significant implications for fundamental rights and public interests.
The primary function of the Kairos AI integration in the Giggle app was to analyze selfies to identify the gender of users for verification purposes.
The reports provide clear, consistent details about the app's functionality, the specific AI vendor (Kairos), the nature of the bias (Eurocentric and trans-exclusionary), and the privacy policy concerns. The scientific limitations of the facial recognition software are backed by a cited MIT study.
None. The primary dispute is ideological (the founder's defense of the exclusion) rather than factual regarding how the AI functioned.
Technology
Biased Facial Recognition
Kairos AI misgendered darker-skinned females 22.5% of the time.
Flawed Bone Structure Proxy
Using facial bone structure as a biological proxy for gender is inaccurate.
Easily Bypassed Filter
Users could bypass rejection by simply submitting another selfie.
Data Inputs
Eurocentric Training Datasets
AI training data favored lighter skin tones and Eurocentric facial features.
Single Selfie Input Limitation
Verification relied on a single selfie, leading to high classification error.
Human Factors
Transphobic Ideology of Founder
Founder intentionally chose to exclude trans women from the platform.
User Frustration and Backlash
Users targeted the app with negative reviews due to unfair exclusion.
Process and Methods
Inadequate Algorithmic Testing
The app launched despite known high error rates for diverse demographics.
Arbitrary Manual Interventions
The CEO manually banned users based on social media callouts.
Intrusive Data Collection
Privacy policy allowed sharing sensitive biometric data with third parties.
Management
Active Exclusion Policy
Management explicitly designed onboarding to gatekeep and exclude trans women.
Dismissal of Algorithmic Bias
Management dismissed concerns about racism and bias in facial analysis.
Regulatory Environment
Weak App Store Policy Enforcement
App stores restored the app despite complaints of bias and exclusion.
Lack of Biometric Regulation
Absence of laws restricting biometric gender categorization on social apps.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
