This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
ProPublica conducted an investigation into Sound Intelligence's aggression detection software, finding that it frequently misidentifies benign sounds like laughter, coughing, and cheering as aggressive behavior. The system, which is deployed in schools, hospitals, and other public facilities, relies on machine learning to flag vocal strain, but often fails to distinguish between enthusiasm and genuine threats. These false positives raise concerns about the reliability of using such algorithms for public safety and the potential for unnecessary security responses.
Sound Intelligence's "aggression detection" algorithm deployed by schools reportedly contained high rates of false positive, misclassifying laughing, coughing, cheering, and loud discussions.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
Schools and hospitals relied on the system for critical security and safety monitoring, despite its inability to reliably distinguish aggression from normal human behavior.
The system demonstrated a significant lack of robustness, frequently generating false positives for benign sounds like coughing and failing to detect actual screaming.
The aggression detection algorithm exhibited unequal performance across demographic groups, specifically showing a higher rate of false positives for female students due to their higher-pitched voices.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The failures of the aggression detection system to accurately classify sounds occurred post-deployment due to unintentional algorithmic limitations.
Added categories peak at 2: Minor (Psychological Harm, Indirect)
The Sound Intelligence aggression detection system's high false-positive rates, failure to detect actual threats, and gender bias in schools and hospitals represent a minor national security concern. While highlighting vulnerabilities in public space AI surveillance and civil rights issues, the impact remains localized and manageable through standard procurement and security procedures.
The system operates as an Assistant because it is designed to augment human security capabilities by providing real-time alerts of potential aggression, but it requires human oversight and decision-making to verify the threat and take action.
Risk Level 2: High Risk. The system is deployed in educational institutions (schools) and healthcare facilities (hospitals) to monitor safety and security, which directly impacts fundamental rights and public interests.
The primary goal of the system is to detect verbal aggression and vocal distress to alert security of potential threats, operating as a smart surveillance tool.
The ProPublica reports provide extensive empirical testing data, methodology, and direct interviews with the AI developer's CEO, school administrators, and hospital security chiefs, offering a highly detailed and credible account of the system's performance and real-world impacts.
The developer argues the system is a valuable early warning system despite its imperfections, whereas critics view it as an unreliable and invasive surveillance tool.
Technology
Context-Blind Algorithm
Algorithm uses audio features without understanding words, meaning, or context.
Acoustic Feature Correlation
Algorithm equates aggression with high pitch and rough or strained vocal tones.
Biased Feature Sensitivity
Higher-pitched female voices are more prone to triggering false positives.
Data Inputs
Unrepresentative Training Data
Training data lacked high-pitched shrieks or school-specific vocalizations.
Incorrect Data Labeling
Some shrieks were labeled as non-aggressive due to similarity to baby cries.
Out-of-Context Calibration
Calibrated on European pub data but used in schools and hospitals.
Human Factors
Overreliance on Automation
Users substitute algorithmic alerts for human observation and judgment.
Misunderstanding System Limits
School officials expected the device to detect quiet, non-strained aggression.
Process and Methods
Inadequate Local Calibration
Fine-tuning sensitivity to stop false alarms increases risk of false negatives.
Flawed Performance Testing
System wasn't comprehensively tested across diverse demographic voice ranges.
Management
Misleading Marketing Claims
Device marketed as school safety solution despite known failure modes.
Premature Commercialization
Selling system to schools and hospitals before resolving high error rates.
Regulatory Environment
Lack of Algorithmic Auditing
No standards require independent auditing of public safety algorithms.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
