This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A Black student experienced difficulty using remote-proctoring software for a lab quiz because the system failed to recognize her face. This incident highlights a broader issue where facial detection software used in educational settings frequently fails to recognize non-white skin tones, leading to students being unfairly flagged to their teachers.
A Black student's face was not recognized by the remote-proctoring software during check-in of a lab quiz, causing her to excessively change her environments for it to work as intended.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
The facial recognition software exhibits unequal performance across demographic groups, failing to recognize non-white skin tones significantly more often than white skin tones.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the AI-based facial detection system failing to recognize a student's face post-deployment, which was an unintentional outcome of the software's design.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
An incident involving facial recognition bias in remote-proctoring software highlights ongoing systemic issues with algorithmic fairness in educational tools. While representing a significant civil rights and educational equity concern, the incident carries negligible direct implications for national security.
The facial detection software acts as a tool designed to perform the specific, well-defined function of verifying a user's identity via camera input during tests.
High Risk: The system is used in an educational setting ('Testing and Recognition Software that is essential for students across the world to use for tests and take classes'), which falls under AI systems affecting access to education and vocational training.
The software is described as facial detection and recognition software used for remote-proctoring and testing, which serves to identify students and detect potential cheating.
The report clearly outlines the specific failure mode (facial recognition failing on darker skin tones) and the context of use (educational testing). While the specific software name is omitted, the nature of the AI safety risk and its real-world impact are unambiguous.
None. The incident is a classic case of demographic bias in computer vision.
Technology
Biased Facial Detection
The software fails to detect or recognize non-white skin tones.
Inadequate Algorithm Design
Algorithms are optimized primarily for lighter skin tones.
Data Inputs
Unrepresentative Training Data
Training datasets lack sufficient images of people of color.
Human Factors
Overreliance on Automated Flags
Teachers trust software alerts without verifying student presence.
Process and Methods
Deficient Software Testing
Testing protocols failed to evaluate performance across diverse groups.
Management
Negligent Procurement
Schools acquire essential software without vetting for algorithmic bias.
Regulatory Environment
Lack of Bias Standards
No regulations mandate demographic bias testing for school software.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
