This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A Florida teenager was wrongfully accused of academic dishonesty by Honorlock, a remote proctoring service that utilizes Amazon Rekognition for facial detection. The system flagged the student for looking away from her screen, which the student explained was a natural behavior for thinking. Despite her explanation, the college upheld the accusation, leading to a failing grade on the exam and a permanent warning on her record.
Broward College’s use of remote proctoring system and reliance on its flagging algorithm allegedly led to a wrongful accusation of academic dishonesty in a biology exam of a Florida teenager.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
Educators overrelied on the automated flags generated by the AI system, treating them as definitive proof of cheating without proper critical evaluation.
The remote proctoring system utilizes Amazon Rekognition, which has documented lower accuracy and higher bias when analyzing darker-skinned females, directly impacting the Black female student who was also misgendered by the tool.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was initiated by the Honorlock AI system's automated flagging of benign human behavior, which was an unexpected outcome of its deployment.
Added categories peak at 2: Minor (Psychological Harm, Direct)
A Florida teenager was falsely accused of cheating by Honorlock proctoring software using Amazon Rekognition, highlighting ongoing concerns about demographic bias and privacy in commercial AI. While raising important civil rights and ethical questions, the incident has negligible direct implications for national security.
The Honorlock system operates as a tool that performs specific, well-defined functions such as facial detection and keystroke monitoring on direct video/audio inputs, producing flags based on pre-set thresholds without autonomous planning.
Risk Level 2 (High Risk): The AI system is used in an educational setting ('remote proctoring companies offered web browser extensions... to prevent the temptation to cheat') where its outputs directly affect student evaluation and academic standing, which falls under educational systems affecting access to education.
The primary function of Honorlock is to detect cheating during remote exams, which it accomplishes in part by using Amazon Rekognition for facial detection and tracking.
The report provides a highly detailed account of a specific student's experience, backed by direct quotes, a review of the video evidence, and technical details regarding the underlying AI model (Amazon Rekognition) and its documented biases.
The school framed the issue as a failure to comply with test-taking directions (looking away) rather than an algorithmic error, though the automated flag was the sole trigger for the investigation.
Technology
Ambiguous Face Detection Flags
Rekognition flags normal behaviors like looking down as suspicious.
Potential Algorithmic Bias
Face detection tools perform less reliably on darker-skinned females.
False Face Detection Triggers
Software historically flagged faces in posters or photos as extra people.
Data Inputs
Lack of Contextual Video Data
The algorithm only analyzes visual cues without understanding human behavior.
Biased Training Datasets
Underlying facial models perform poorly on diverse skin tones and genders.
Human Factors
Confirmation Bias by Educators
Teachers treated algorithmic flags as definitive proof of academic dishonesty.
Normal Human Movements Penalized
Students looking down to think are falsely categorized as cheating.
Lack of Technical Literacy
Educators lack training to properly evaluate flagged automated alerts.
Process and Methods
Inadequate Video Review Process
Faculty did not properly investigate the context of the 50-second clip.
Over-reliance on Automated Alerts
Schools used automated flags directly to issue disciplinary actions.
Asynchronous Course Distance
Fully remote and asynchronous classes reduced student-teacher interaction.
Management
Shift of Responsibility to Schools
Honorlock delegates the challenging task of interpreting flags to educators.
Prioritizing Automation over Review
Schools adopted fully automated proctoring without hiring human proctors.
Regulatory Environment
Absence of Student Privacy Laws
Lack of regulations governing invasive remote proctoring surveillance.
No Algorithmic Audit Standards
No regulatory standards require independent bias audits for proctoring tools.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
