This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
SN Technologies is accused of misleading the Lockport City School District regarding the accuracy and racial bias of its AEGIS facial recognition and weapon detection system. Documents reveal the system frequently misidentified objects as weapons and showed significant racial bias in facial recognition, causing distress among parents and students. The incident highlights concerns over the deployment of unverified AI surveillance tools in educational settings.
SN Technologies allegedly misled Lockport City Schools about the performance of its AEGIS face and weapons detection systems, downplaying error rates for Black faces and weapon misidentification.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
The weapon detection system frequently failed to perform reliably, mistaking broom handles for guns and requiring multiple recalibrations.
The AEGIS system demonstrated significant racial bias, misidentifying Black students at much higher rates (4x for Black men and 16x for Black women) than white students, resulting in unequal performance across demographic groups.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The primary risks and harms arose from the AI system's biased facial recognition performance and false weapon detections after it was deployed in January 2020.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
The deployment of the AEGIS surveillance system in New York schools highlights civil rights and privacy concerns from biased AI tools. While the system exhibited significant racial bias and high false-positive rates, its impact is localized to school administration and civil rights litigation, presenting minimal direct threat to national security.
The AEGIS system is designed to scan video feeds and generate alerts for human verification, augmenting human security capabilities rather than taking autonomous action.
High Risk: The system is a facial recognition and surveillance tool deployed in an educational institution, which falls under high-risk applications affecting fundamental rights and safety.
The system was deployed to identify individuals on a hot list using facial recognition and to detect weapons within school buildings.
The report is highly detailed, citing internal emails, independent audit reports, and statements from NIST scientists, school board members, and the developer.
The developer argues the system is 99.9% accurate and that the issues were minor calibration adjustments, contrasting with the critics' claims of systemic failure.
Technology
High Racial Bias in Algorithm
The licensed id3 algorithm misidentified Black students at much higher rates.
Poor Object Discrimination
The computer vision system regularly mistook broom handles for weapons.
Sensitivity to Camera Disturbance
School construction disturbed cameras, requiring frequent recalibrations.
Data Inputs
Biased Training Data
Underlying facial recognition model performed worse on non-white students.
Obscured Visual Inputs
Pandemic mask-wearing rendered the facial recognition inputs ineffective.
Uncalibrated Weapon Profiles
Lack of diverse negative weapon training examples led to false gun alerts.
Human Factors
Vendor Misrepresentation of Bias
The vendor CEO misled school officials about the algorithm's actual bias rates.
Fear of Armed Police Escalation
Parents feared false alerts would trigger aggressive, tragic police reactions.
Overreliance on Automated Alerts
Initial system design used automated alerts before adding human verification.
Process and Methods
Inadequate Independent Auditing
The district relied on vendor claims before an audit revealed the true bias.
Lack of Pre-deployment Testing
The system was deployed in schools without resolving basic detection errors.
Flawed Accuracy Calculations
NIST scientists described vendor accuracy claims as completely nonsensical.
Management
Procurement Based on Misinformation
School board purchased the system based on misleading vendor claims.
Failure to Address Stakeholder Fears
District management ignored parent and civil liberties groups concerns.
Inadequate Risk Assessment
Management failed to assess risks of false weapon alerts triggering police.
Regulatory Environment
Weak State Oversight
NYSED abruptly approved the system after initially opposing its deployment.
Absence of School AI Bans
Lack of timely legislation allowed districts to deploy unvetted AI systems.
Unregulated Funding Allocation
Smart Schools funding was approved for surveillance without safety guidelines.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
