This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Researchers at the University of North Carolina at Wilmington compiled a dataset of 10,000 images of transgender individuals scraped from YouTube without consent to train facial recognition systems, which was later found to be distributed to other institutions. Separately, Mitek's identity verification system failed to correctly process non-binary gender markers on California IDs, potentially preventing users from accessing financial services. Both incidents highlight the ethical risks of biometric data collection and algorithmic bias against marginalized communities.
YouTube videos of transgender people used by researchers to study facial recognition during gender transitions were used and distributed without permission.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
Also classified as
Mitek's identity verification system failed to correctly process non-binary gender markers, resulting in unequal system performance and exclusion for non-binary individuals.
The primary incident involves researchers compiling and distributing highly sensitive personal transition videos of transgender individuals without consent, and leaving them in an unprotected Dropbox.
Human
Due to a decision or action made by humans
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risks and harms were caused by human decisions, including researchers scraping data without consent and developers introducing a coding error in the ID scanning software.
Added categories peak at 2: Minor (Psychological Harm, Direct)
The incidents involve unauthorized scraping of transgender individuals' biometric data for military/intelligence-funded research distributed internationally, alongside algorithmic bias in commercial identity verification. While raising significant ethical, privacy, and civil rights concerns, the direct national security impact remains minor and localized.
The facial recognition algorithms and Mitek's digital identity verification scanner operate as tools that perform specific, deterministic functions on direct inputs without maintaining an internal state or executing autonomous plans.
High Risk: The report describes facial recognition and biometric identity verification systems, which are classified as high-risk under the EU AI Act due to their use in law enforcement, biometrics, and access to essential private services like banking.
The primary goal of the UNC Wilmington research was to evaluate and improve facial recognition systems, specifically tracking facial changes during hormone replacement therapy.
The reports provide detailed, consistent accounts of the UNC Wilmington dataset's creation, distribution, and the privacy concerns raised by participants and researchers. The Mitek incident is also clearly described with specific technical details regarding the barcode scanning error.
The Mitek incident could be viewed purely as a software database bug rather than an AI safety failure, though its impact on biometric identity verification aligns it with algorithmic bias.
Technology
Algorithmic fragility to HRT
Facial recognition systems struggle to adapt to HRT facial changes.
ID scanning software errors
Mitek systems failed to authenticate non-binary IDs due to coding errors.
Data Inputs
Scraping without consent
YouTube videos were scraped for images without the creators' permission.
Insecure dataset storage
The dataset was stored in an unprotected Dropbox folder accessible via URL.
License agreement violations
Standard YouTube licenses prohibiting redistribution were ignored.
Human Factors
Disregard for participant privacy
Researchers prioritized technical challenges over participant privacy.
Transphobic research assumptions
Hypothesized that trans people use HRT to evade border surveillance.
Process and Methods
Bypassing ethical review boards
The researcher did not seek institutional ethical approval for the dataset.
Uncontrolled dataset distribution
The dataset was shared with 15 institutions globally without oversight.
Inadequate contact for consent
Researchers failed to contact or obtain consent from all participants.
Management
Inadequate research oversight
University failed to monitor the creation and sharing of the dataset.
Unmonitored government funding
Federal agencies funded biometric research without evaluating ethical risks.
Regulatory Environment
Lack of public data regulations
Absence of clear regulations on scraping public social media for AI training.
Permissive default licensing
Default CC licenses allowed massive scraping of personal images.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
