This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
An AI-powered gun detection system operated by Omnilert at Kenwood High School misidentified a student's bag of chips as a firearm. This false positive triggered a high-stakes police response, resulting in the student being handcuffed and held at gunpoint. The incident has prompted calls from local officials for a review of school security protocols and the reliability of the AI system.
A purportedly AI-powered gun detection system at Kenwood High School in Baltimore County, Maryland, reportedly misidentified a student's empty Doritos bag as a firearm. Armed police reportedly detained and handcuffed the student before determining there was no weapon. The AI system, operated by Omnilert, issued the false alert, leading to public outcry and an official review of school security protocols. Omnilert reportedly expressed regret for the incident.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
School staff acted on the AI's initial alert and initiated a high-stakes police response before fully verifying the system's canceled status.
The AI system failed to perform robustly under real-world conditions by misidentifying a common object (a bag of chips) as a lethal weapon, leading to severe real-world consequences.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was triggered by a false positive error from the deployed Omnilert AI system, which was an unexpected outcome of its threat detection goal.
Added categories peak at 2: Minor (Psychological Harm, Indirect)
An AI gun detection system's false positive misidentified a student's bag of chips as a firearm, triggering a high-stakes police response where a student was held at gunpoint. While raising local civil rights and public safety AI reliability concerns, the incident has negligible national security implications.
The Omnilert system acts as an assistant by analyzing surveillance footage to flag potential threats for human verification, rather than taking independent action. As noted by school officials, 'the system is looking for certain elements, and then humans have to verify them.'
High Risk: The AI system is used for safety and threat detection in an educational institution, which directly interfaces with law enforcement response, significantly impacting the safety and fundamental rights of students.
The system's primary function is to detect guns (threats) using school security cameras (surveillance) to prevent violence.
The reports provide highly consistent, detailed accounts of the incident, including direct quotes from the affected student, school administrators, local councilmembers, and the AI developer. The sequence of events is clear and uncontested.
None. The incident is clearly a false positive detection leading to an unwarranted police response.
Technology
AI Computer Vision Error
The AI system mistook a Doritos bag for a firearm based on shape.
Vision-Based Limitations
The system can only identify what is visible to the security cameras.
Data Inputs
Misleading Visual Input
The student holding the chip bag closely resembled a gun pose.
Human Factors
Communication Oversight
The principal did not realize the alert had already been canceled.
Aggressive Police Response
Officers responded with drawn guns and handcuffed an unarmed student.
Process and Methods
Flawed Alert Cancellation Flow
The canceled alert did not stop the principal from contacting police.
Inadequate Verification Process
The multi-step human verification failed to prevent police escalation.
Management
Lack of Clear Procedures
School board lacked clear protocols for handling AI gun alerts.
Inadequate System Safeguards
Lack of safeguards to prevent false alarms from escalating to police.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
