This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A month-long pilot of Evolv's AI-powered weapons scanners in NYC subway stations failed to detect any firearms while generating 118 false positives. The incident highlights concerns regarding the efficacy of the technology, which the FTC has alleged was marketed using misleading claims about its ability to distinguish weapons from harmless metallic objects.
NYC implemented an AI enabled weapons scanner for a month-long pilot with limited success. Despite not finding any weapons during the September 2024 testing phase, there were 118 false positives in which a person was searched under suspicion of carrying a weapon with no actual gun detections.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
Evolv engaged in misleading marketing practices, overstating the AI's capabilities to sell the scanners at up to five times the rate of standard metal detectors.
City officials deployed the technology in a critical public transit environment despite explicit warnings from the developer's CEO that subways were an unsuitable use case.
The Evolv scanner failed to perform reliably under subway conditions, resulting in a high false positive rate and zero weapon detections, demonstrating a clear lack of capability and robustness.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was primarily caused by the AI system's failure to perform reliably in a deployed environment, generating high rates of false positives and failing to detect actual weapons.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
A pilot program of Evolv's AI weapons scanners in the NYC subway system failed to detect any firearms while generating 118 false positives. The incident highlights minor national security concerns regarding the reliability of AI in critical infrastructure security and the risks of misleading marketing influencing public procurement.
The AI system operates independently in real-time, analyzing passengers as they pass through the scanner and making autonomous decisions to trigger alerts based on learned patterns of metallic objects, without requiring direct human execution for each scan.
Risk Level 2 (High Risk): The report describes an AI system used in law enforcement applications and critical public infrastructure (subway transit), which are explicitly classified as High Risk under the EU AI Act.
The system's primary advertised and deployed function is to scan individuals to detect weapons (threats) in public spaces and schools.
The reports provide official statistics from the NYPD (2,749 scans, 118 false positives) and clear details regarding the FTC settlement with Evolv. The facts are consistent across both sources, leaving little ambiguity about the system's performance and the regulatory actions taken.
None
Technology
Subway Electromagnetic Interference
Railway interference degraded the accuracy of the scanner sensors.
Basic Metal Detection Limitations
The underlying tech functions as a metal detector, not a weapon sensor.
Flawed AI Classification
AI algorithms failed to distinguish weapons from harmless metal objects.
Data Inputs
Lack of Subway-Specific Data
System was not trained on data reflecting subway environmental noise.
Inadequate Signal Filtering
Sensor inputs failed to filter out common metallic items in transit.
Unrefined Sensor Input Data
Sensor inputs were triggered by harmless personal metallic items.
Human Factors
Political Overconfidence
The NYC Mayor pushed deployment despite technical warnings from the CEO.
Operator Alert Fatigue
Over 100 false alarms created unnecessary searches and officer burden.
User Misreliance on Marketing
Buyers relied on overstated marketing claims rather than empirical data.
Process and Methods
Inadequate Pilot Evaluation
The 30-day pilot lacked rigorous pre-deployment testing and metrics.
Deceptive Marketing Practices
Sales teams overstated weapon detection accuracy and capabilities.
Lack of Independent Validation
No third-party testing was done to verify claims before subway trial.
Management
Prioritizing Hype Over Safety
Company and city officials prioritized high-tech marketing over accuracy.
Ignoring Internal Warnings
Management deployed tech despite the CEO stating subways are a bad use case.
Misleading Investor Relations
Executives overstated capabilities, leading to shareholder lawsuits.
Regulatory Environment
Lax Pre-Market AI Oversight
No regulatory standards required proof of AI efficacy before public use.
FTC and SEC Interventions
Regulatory bodies acted post-deployment rather than preventing false claims.
Lack of Public Transit Standards
No specific safety or accuracy mandates exist for transit AI scanners.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
