This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Discord's automated image-moderation system incorrectly flagged over 8,000 benign images as prohibited content, resulting in wrongful account bans. A bug in the system prevented these accounts from being automatically restored after being cleared by staff. Discord has since reinstated all affected users.
Discord reportedly wrongfully banned more than 8,000 accounts after its automated image-moderation workflow flagged harmless uploads, including grids, spreadsheets, chessboards, and game images, as prohibited material. Discord said a bug converted temporary upload restrictions into account bans and prevented cleared accounts from being restored automatically. The company later reinstated the affected accounts.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
The automated safety system failed to perform reliably, incorrectly flagging benign grid-like images as harmful and executing erroneous bans due to a software glitch.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was triggered by an operational AI-driven content moderation system post-deployment, which unintentionally flagged benign images and incorrectly banned users due to a system bug.
An unintentional software bug in Discord's automated image-moderation system caused wrongful bans of over 8,000 accounts. The incident was quickly resolved and accounts were reinstated, presenting negligible national security implications across all assessed categories.
The automated safety system operated independently to scan uploaded images, match them against known harmful material, and execute real-time account bans without human intervention.
Minimal or No Risk: The system is an automated content moderation filter, which is analogous to 'spam filters' and other simple applications that pose low or negligible risks to society under the EU AI Act.
The AI safety system is explicitly designed to moderate content by matching uploaded images against known harmful material to flag violations.
The incident reports from Discord support and tech journalism provide clear, consistent, and explicit details about the nature of the bug, the number of users affected (around 8,200), and the resolution of the incident. There is little ambiguity about what transpired.
The incident could be interpreted purely as a software engineering bug rather than an AI failure, but since it involves an automated image-matching safety system, it is classified as an AI robustness failure.
Technology
False Positive Matching Glitch
System incorrectly matched benign grid images with known harmful content.
Immediate Account Ban Bug
Glitch caused immediate account bans instead of temporary upload blocks.
Failed Ban Reversal Bug
Bug prevented account bans from being automatically lifted after staff review.
Data Inputs
Grid-like Image Inputs
Chessboards and textures triggered false positives in detection system.
Benign Image Variations
Other benign images since May triggered false positive matches.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
