This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Proctorio, an AI-based remote proctoring software, faced widespread backlash from students and universities due to concerns regarding privacy, accessibility, and racial bias. Reports indicated that the software's facial detection algorithms struggled to recognize students with darker skin tones, leading to potential false accusations of academic dishonesty. Consequently, several universities, including the University of Illinois Urbana-Champaign, decided to discontinue their use of the software.
Proctorio's remote-testing software were reported by students at the University of Illinois Urbana-Champaign for issues regarding privacy, accessibility, differential performance on darker-skinned students.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
The software monitored students' private homes, webcams, and keystrokes, violating their expectation of privacy.
The system failed to perform robustly under varying conditions, flagging students with glasses, headscarves, or disabilities.
The proctoring software's facial detection algorithm performed significantly worse for Black students, failing to recognize them over half the time.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The algorithmic bias and accessibility failures were unintentional outcomes of the deployed AI system's facial detection capabilities.
Added categories peak at 2: Minor (Psychological Harm, Direct)
An AI-based remote proctoring software, Proctorio, exhibited algorithmic bias against students of color and violated student privacy during remote exams. While causing significant public backlash and academic disruption, the incident has negligible direct national security implications, representing a localized civil rights and consumer privacy concern managed through institutional policy and legal channels.
The software acts as an Assistant because it monitors students and flags suspicious behavior to augment the instructor's capability to detect cheating, while requiring human oversight to make final decisions.
High Risk: The proctoring software is an educational system affecting access to education, as it has the power to accuse students of cheating and flag them to instructors, which is classified as a High Risk AI system under the EU AI Act.
The primary goal of Proctorio is cheating detection during online exams, utilizing facial detection and gaze tracking to monitor students.
The reports provide consistent, detailed accounts from multiple reputable sources regarding the software's functionality, the specific biases identified, and the resulting student backlash and university contract terminations.
Proctorio argues the software is a neutral tool and that any detection failures are due to environmental factors like lighting rather than algorithmic bias.
Technology
Flawed Facial Detection Code
Relies on open-source OpenCV which has a history of racial bias.
Inflexible Algorithmic Rules
Flags normal behaviors like stimming or wearing glasses as suspicious.
Cognitive Bias in Tracking
Gaze tracking flags ADHD behaviors as potential cheating.
Data Inputs
Biased Training Data
Algorithms perform poorly on diverse skin tones due to biased training sets.
Environmental Variables
Poor lighting and webcam quality degrade facial detection performance.
Hardware Requirements
Requires high bandwidth and webcams, disadvantaging some students.
Human Factors
High Student Test Anxiety
Constant monitoring and invasive surveillance cause severe psychological stress.
Unfair Living Conditions
Assumes all students have access to quiet, private testing environments.
Over-reliance on AI Flags
Instructors may trust automated flags without conducting proper manual reviews.
Process and Methods
Proprietary Black Box Software
Lack of source code auditing prevents independent verification of fairness.
Inadequate Feedback Loop
Falsely flagged students face difficult processes to clear their names.
Automatic Suspect Flagging
System automatically flags students to instructors based on arbitrary signals.
Management
Suppression of Critics
Filing SLAPP lawsuits against researchers who highlight software flaws.
Neglecting Equity and Access
Deploying software without assessing impact on marginalized students.
Lack of Operational Transparency
Management failed to provide clear details on how the software operates.
Regulatory Environment
Abuse of DMCA Provisions
Company used copyright takedowns to silence critical security researchers.
Lack of AI Proctoring Standards
No clear regulatory oversight governing the use of AI in education.
Lack of Privacy Protections
Inadequate legal safeguards for student data privacy in remote settings.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
