This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Students at the University of Toronto and other institutions reported that remote exam proctoring software, specifically ProctorU, exhibited racial bias in its facial recognition systems. The software frequently failed to verify the identities of BIPOC students, leading to increased stress and lost exam time. These incidents highlight systemic issues with the deployment of AI-based monitoring tools in educational settings.
An exam monitoring service used by the University of Toronto was alleged by its students to have provided discriminatory check-in experiences via its facial recognition's failure to verify passport photo, disproportionately enhancing disadvantaging stress level for BIPOC students.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
Educational institutions overrelied on automated proctoring tools for high-stakes testing despite known equity and performance limitations.
The facial recognition technology lacked the robustness to perform reliably across different skin tones and lighting conditions.
The remote proctoring software exhibited unequal performance across demographic groups, specifically failing to recognize BIPOC students' faces and passports compared to white students.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the ProctorU AI system's biased facial recognition algorithm failing to verify BIPOC students post-deployment, which was an unintentional outcome of its design.
Added categories peak at 2: Minor (Psychological Harm, Direct)
Remote proctoring software ProctorU exhibited racial bias, failing to verify the identities of BIPOC students at North American universities. While raising important civil rights and ethical concerns regarding AI deployment in education, the incident has negligible direct implications for national security.
The AI system functions as an Assistant, designed to augment human proctors by performing automated identity verification and flagging potential cheating, while leaving final decisions to human oversight. The report notes that 'ProctorU uses a diverse group of trained human proctors, assisted by technology, to safeguard exam integrity'.
Risk Level 2: High Risk. The system is an educational assessment tool used in higher education, which falls under High Risk as it affects students' access to education. The report notes that the software's inability to recognize facial features 'temporarily barred them from accessing the software' and taking their exams.
The primary function of the AI system described is to recognize and verify the faces and identities of students (Face Recognition) to prevent academic dishonesty during remote exams (Cheating Detection).
The reports provide clear, first-hand accounts from affected students (Chelsea Okankwu, Maame Adjoa) and corroborate these with statements from university officials, civil rights attorneys, and US Senators. The technical failure (facial recognition bias against BIPOC individuals) is well-documented and aligned with established AI research.
None
Technology
Bias in Facial Recognition Tech
Software struggles to identify darker skin tones, causing check-in failures.
Automated Proctoring Failures
AI systems fail to recognize passports or facial features of BIPOC students.
Data Inputs
Underrepresentative Training Data
Datasets used to train AI models underrepresent BIPOC individuals.
Human Factors
Increased Student Stress
Verification failures and intense check-in requirements cause pre-exam anxiety.
Process and Methods
Intrusive Check-in Procedures
Requirements like 360 room scans and computer sweeps enhance test anxiety.
Lack of Alternative Assessments
Heavy reliance on high-stakes exams instead of project-based learning.
Management
Decentralized Decision-making
Individual faculties decide on software usage without central oversight.
Continued Use of Biased Vendors
Partnerships with ProctorU and Examity continued despite known software biases.
Regulatory Environment
Lack of University-wide Policy
UofT has no centralized regulation banning or strictly limiting e-proctoring.
Non-binding Equity Guidelines
Official advice to limit proctoring is framed as a recommendation, not a rule.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
