This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A user reported that FaceApp and the 'how-old' website provided inconsistent gender and age classifications based on minor physical changes, such as eyebrow thickness or wearing glasses. The user described how these algorithmic outputs triggered gender dysphoria and distress, leading them to alter their personal appearance to conform to the AI's classification patterns.
FaceApp’s algorithm was reported by a user to have predicted different genders for two mostly identical facial photos with only a slight difference in eyebrow thickness.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
The user relied on consumer entertainment classification apps for personal gender validation, which triggered emotional distress when the systems failed to perform robustly.
The AI systems exhibited unequal performance and accuracy in gender classification that is dependent on demographic group presentation, leading to the alienation and distress of a gender-nonconforming user.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The harm was caused by the classification decisions of the deployed AI systems, which produced unexpected negative psychological outcomes for the user.
Added categories peak at 2: Minor (Psychological Harm, Direct)
A consumer experienced gender dysphoria and altered their appearance due to inconsistent gender and age classifications by FaceApp and how-old. These classification errors in consumer entertainment apps present negligible national security implications across all assessed categories.
The AI systems (FaceApp and the how-old site) operate as tools that perform specific, well-defined classification functions on direct image inputs without context retention or autonomous planning.
Minimal or No Risk: The AI systems involved are consumer entertainment applications used for photo editing and simple age/gender estimation, which carry no additional regulatory obligations under the Act.
The primary function of the FaceApp feature described is to identify and classify the gender of the person in the uploaded image, while the how-old site estimates age and gender.
The report is a direct, first-hand account detailing the user's specific interactions with the AI systems and the resulting psychological impact. The behavior of the systems and the user's response are clearly described without conflicting details.
The incident could be interpreted purely as a user experience issue rather than an AI safety failure, but the psychological harm and behavioral modification point to a safety risk in demographic classification.
Technology
Simplistic Heuristics
Model relies on superficial features like brow thickness to classify gender.
Fragile Vision Models
Minor changes like wearing glasses drastically alter gender and age outputs.
Data Inputs
Biased Training Data
Dataset correlates thick eyebrows with males and thin eyebrows with females.
Poor Representation
Underrepresentation of non-binary or transgender features in training data.
Human Factors
Validation Seeking
User relies on AI apps for gender affirmation, triggering dysphoria.
Behavioral Modification
User alters physical appearance like overplucking brows to satisfy the AI.
Process and Methods
Inadequate Bias Testing
Lack of robust testing for demographic variations and superficial features.
No Safety Guardrails
Lack of warnings about model limitations on sensitive classifications.
Management
Neglecting Mental Risks
Developers prioritize viral engagement over the mental health risks of users.
Regulatory Environment
Absence of Bias Standards
No regulatory standards governing gender classification or bias in consumer AI.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
