This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A coordinated neo-Nazi network is using generative AI to create and spread extremist propaganda on TikTok, including dehumanizing caricatures and AI-translated speeches of Nazi leaders. This content is designed to modernize Nazi ideology and evade platform moderation. TikTok's recommendation algorithm further amplifies this content, recommending it to users and facilitating the growth of extremist networks, which are also using the platform for real-world recruitment and coordination of hate speech.
A coordinated neo-Nazi network on TikTok used AI-generated media, including Hitler speeches, to spread Nazi propaganda and extremist content, violating TikTok’s hate speech policies. The network evaded platform moderation through coded language, imagery, and music, with some accounts accumulating millions of views. TikTok’s algorithm further amplified the reach of this content, despite community guidelines.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
A coordinated network of malicious actors is using generative AI and platform algorithms to conduct large-scale propaganda and recruitment campaigns.
TikTok's automated content moderation systems failed to detect and remove flagrant violations of hate speech and extremist content.
The incident primarily involves users being exposed to harmful, abusive, and extremist content, specifically Nazi propaganda and hate speech, amplified by TikTok's recommendation algorithm.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk is amplified by TikTok's recommendation algorithm, which unintentionally promotes extremist content as an unexpected outcome of optimizing for user engagement.
Added categories peak at 3: Substantial (Epistemic Harm, Indirect)
A coordinated neo-Nazi network leveraged generative AI and TikTok's recommendation algorithm to disseminate translated extremist speeches and propaganda at scale. The incident highlights an evolved capability in information warfare and algorithmic exploitation, posing substantial threats to societal stability and human rights through automated radicalization.
The recommendation algorithm operates independently to curate and serve content to users' For You Pages based on real-time interaction patterns without explicit human planning for each recommendation.
Limited Risk: The report describes the use of generative AI to create deepfakes (AI-translated speeches and caricatures), which falls under transparency obligations for AI-generated content.
The primary driver of the content's reach is TikTok's recommendation algorithm, while generative AI translation tools were used to translate historical speeches.
The report by ISD Global is highly detailed, providing specific methodologies, account counts, view statistics, and clear examples of how both generative AI and TikTok's recommendation algorithm functioned in this incident. The role of the AI systems is explicitly documented through dummy account testing.
The incident could be viewed primarily as a human moderation failure rather than an AI safety failure, though the active algorithmic amplification by the FYP recommendation engine makes it a clear AI safety issue.
Technology
Algorithmic Amplification
The FYP algorithm prioritizes engagement, pushing extremist content to users.
Generative AI Propaganda
AI-generated translations modernize historical Nazi speeches to evade filters.
Audio Moderation Evasion
Extremist sounds and music bypass automated speech and audio filters.
Data Inputs
Coded Language and Emojis
Nazi-coded emojis and numbers are not flagged as hate speech data inputs.
Obfuscated Image Thumbnails
Innocuous thumbnails trick visual classifiers before transitioning to hate.
User Engagement Signals
Superficial views and clicks train the algorithm to recommend more hate.
Human Factors
Cross-Platform Coordination
Organized groups coordinate on Telegram to mass-engage and boost videos.
Follow-for-Follow Networking
Users build dense networks to rapidly spread and restore banned accounts.
Inadequate Human Moderation Review
Human reviewers fail to recognize coded Nazi symbolism during manual reports.
Process and Methods
Individual Account Focus
Moderation targets single accounts rather than systemic coordinated networks.
Delayed Ban Enforcement
Accounts remain active for weeks, accumulating millions of views before bans.
Ineffective Warning Systems
Final warnings allow users to transition followers to backup accounts.
Management
Prioritizing Engagement Over Safety
TikTok management designs algorithms to prioritize user watch-time over safety.
Inadequate Moderation Resources
Management fails to invest in tools detecting coordinated network campaigns.
Flawed Policy Enforcement
Systemic failure to enforce community guidelines on reported extremist content.
AI system involved
An AI system was involved, but the reports do not identify it.
Identified by automated extraction; can differ from the alleged developers above.
