This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Sichuan provincial police identified and penalized perpetrators in 10 distinct cases where AI tools were used to generate and spread misinformation. These incidents involved fabricated reports of natural disasters, social unrest, and personal tragedies, designed to attract traffic and generate profit. One perpetrator even developed an automated system that generated over 420,000 fake articles using AI tools.
In Sichuan, AI tools were reportedly used to create and disseminate false reports across various cities, including fabricated stories of landslides, earthquakes, standoffs, accidents, and health-related incidents. These AI-driven rumors misled the public, caused alarm and confusion, and strained local authorities. The alleged perpetrators sought social media engagement and financial gain through clickbait content. The authorities in Sichuan ultimately levied administrative penalties against the perpetrators.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
One perpetrator generated over 420,000 fake articles at scale, significantly polluting the online information ecosystem.
The perpetrators generated these false rumors to gain personal financial advantage through traffic monetization.
The primary issue is the intentional generation and dissemination of false and misleading news articles and images using AI tools, which misled the public.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incidents were caused by human actors who intentionally used deployed AI tools to generate and spread false rumors for personal profit.
Added categories peak at 4: Severe (Epistemic Harm, Inferred)
Chinese domestic actors utilized generative AI tools to fabricate and disseminate over 420,000 fake articles, including false reports of natural disasters and violence, to generate ad revenue. While causing localized public anxiety and social panic, the incident represents a minor national security threat focused on profit-driven misinformation rather than coordinated state-sponsored information warfare.
The AI systems functioned as tools, generating text and images based on direct prompts and inputs from the users to create fake news articles.
Risk Level 3: Limited Risk. The incident involves the use of AI to generate synthetic text and images (deepfakes/fake news), which falls under transparency obligations for AI-generated content.
The AI systems were primarily used to generate fake news articles and social media posts, with some cases also generating fake disaster images to attract traffic.
The report from the Sichuan Provincial Public Security Department is highly detailed, outlining ten specific cases with names of perpetrators, locations, AI tools used, and the exact nature of the rumors generated.
None, as the police investigation clearly established the intentional use of AI to generate rumors.
Technology
AI Text and Image Generators
AI software lowered the technical barrier to generate realistic fake content.
Automated Content Generation Tools
Tools like 'Cuilv Bimo' enabled bulk generation of fake articles.
Realism of AI-Synthesized Media
AI-edited videos and images easily deceived viewers into believing rumors.
Data Inputs
Misappropriated Real-world Media
Real photos and foreign videos were used as inputs to seed AI generation.
Unverified Online Information
Unverified news and rumors were fed into AI tools to synthesize fake stories.
Unfiltered Prompt Inputs
AI tools processed false or misleading prompts without truth verification.
Human Factors
Profit-Driven Motivation
Creators fabricated AI rumors to attract traffic, clicks, and rewards.
Lack of Ethical Awareness
Users generated harmful AI content without considering public panic.
Credulity of Online Audiences
Netizens easily believed and shared realistic AI-generated fake news.
Process and Methods
Automated Batch Posting Systems
Systems automated draft creation and posting workflows for AI rumors.
Synthesis and Editing Methods
Grafting real images onto unrelated AI stories misled the public.
Inadequate Platform Filtering
Platform mechanisms failed to detect and block AI-generated rumors.
Management
Platform Reward Loops
Platform monetization algorithms incentivized sensationalist AI content.
Lack of AI Tool Guardrails
Developers of AI tools failed to implement strict content safety filters.
Volunteer Network Exploitation
Schemes managed large groups of volunteers to distribute AI-generated rumors.
Regulatory Environment
Lagging AI Content Regulations
Regulations on labelling AI-generated content were not fully preventative.
Post-hoc Enforcement
Law enforcement acted after the AI rumors had already spread and caused panic.
Lack of Platform Accountability
Platforms lacked strict penalties for hosting AI-generated disinformation.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
