This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A University of Ottawa research report, supported by the Office of the Privacy Commissioner of Canada, found that AI-based remote exam proctoring tools failed to meet legal standards for consent and privacy. The study highlighted that these tools often produce discriminatory outcomes, disproportionately flagging students based on socio-economic factors such as living conditions, pets, or family structure, rather than actual cheating.
AI proctoring tools for remote exams were reportedly "not conducive" to individual consent for Canadian students whose biometric data was collected during universities' use of remote proctoring in the COVID pandemic.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
The tools collect and process sensitive biometric data, including facial recognition and eye tracking, without obtaining valid, free, and clear consent from students.
The proctoring software's accuracy and effectiveness differ by demographic and socioeconomic groups, disproportionately flagging students from multigenerational homes or specific backgrounds.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The discriminatory outcomes and false cheating flags are caused by the AI systems' technical limitations and biased performance post-deployment, which was an unexpected outcome of pursuing cheating detection.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
The deployment of AI-based remote exam proctoring tools in Canadian universities led to privacy violations and algorithmic bias against lower-income and intersectional students. While raising significant civil rights, privacy, and regulatory compliance concerns, the incident remains localized to the educational sector and does not pose a substantial threat to Canadian national security.
The AI systems independently monitor students in real-time, analyzing biometric and environmental inputs to autonomously flag potential cheating incidents without direct human intervention during the exam.
High Risk: The AI systems are educational evaluation tools that monitor exams and collect biometric data, directly affecting students' access to education and academic standing.
The primary function of these AI systems is to monitor students during exams to detect and prevent cheating using facial recognition, eye tracking, and audio analysis.
The reports are based on a comprehensive academic study from the University of Ottawa supported by the Office of the Privacy Commissioner of Canada. They clearly outline the privacy, consent, and discriminatory issues associated with AI proctoring tools, providing specific examples of failure modes.
The issues could be viewed primarily as a failure of university administration and legal procurement rather than an inherent AI safety failure, though the algorithmic biases are central to the reported harms.
Technology
Facial Recognition Biases
Biometric algorithms exhibit racial, gender, and age discrimination.
Unreliable Detection Algorithms
Algorithms mistakenly flag benign audio-visual cues as cheating.
Intrusive Biometric Tracking
Tracking keystrokes and eye movements poses severe privacy risks.
Data Inputs
Socioeconomic Data Variations
Noisy environments or poor equipment trigger false cheating alerts.
Biometric Data Collection
Collection of highly sensitive biometric data without safeguards.
Cross-Border Data Transfers
Data transfer to US-based servers complicates control over retention.
Human Factors
Forced Student Consent
Students cannot freely refuse consent due to academic pressure.
Lack of Student Control
Students have no control over how their biometric data is retained.
Overreliance on AI Flags
Universities rely heavily on automated flags to judge academic integrity.
Process and Methods
Flawed Consent Gathering
Consent processes do not ensure clear and individual consent.
Inadequate Human Oversight
Lack of human review to verify automated cheating alerts.
Inflexible Testing Conditions
Proctoring methods fail to adapt to diverse student living conditions.
Management
Rapid Software Adoption
Universities rushed implementation during the COVID-19 pandemic.
Outsourcing Vendor Risks
Adopting US-based tools without assessing data protection risks.
Neglecting Socioeconomic Bias
Management failed to assess risks to marginalized student populations.
Regulatory Environment
Inadequate AI Regulations
Existing laws like PIPEDA fail to address AI proctoring risks.
Jurisdictional Fragmentation
Conflict between public university rules and private outsourcing laws.
Lack of Prohibited AI Rules
Absence of clear rules prohibiting high-risk surveillance use cases.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
