This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
In late 2022, researchers at Graphika identified a pro-China influence operation known as 'Spamouflage' that utilized Synthesia's AI video generation platform to create fictitious news anchors for a fake outlet called 'Wolf News.' These deepfake videos were used to disseminate propaganda and critique U.S. policy. While the campaign had limited reach and the videos were of low quality, the incident marked the first known instance of a state-aligned actor using AI-generated video of fictitious people for political disinformation.
Synthesia's AI-generated video-making tool was reportedly used by Spamouflage to disseminate pro-China propaganda news on social media using videos featuring highly realistic fictitious news anchors.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to conduct large-scale disinformation campaigns, malicious surveillance, or targeted and sophisticated automated censorship and propaganda, with the aim of manipulating political processes, public opinion, and behavior.
Also classified as
The AI system was used to generate false personas and misleading news broadcasts to deceive viewers.
Synthesia's content moderation system failed to detect and block the policy-violating political content.
The incident involved a state-aligned influence operation using AI-generated avatars to conduct a coordinated disinformation campaign aimed at manipulating public opinion and political processes.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by human actors who intentionally used a deployed commercial AI video generation platform to create deceptive political propaganda.
Added categories peak at 4: Severe (Epistemic Harm, Inferred)
A pro-China influence operation utilized commercial AI video generation software to create fake news anchors for political propaganda. While the immediate national security impact is minor due to low quality and very low engagement, the incident establishes a significant precedent as the first known state-aligned use of generative AI for political disinformation.
The AI system functions as a tool, where users type a script and the software generates a video of an avatar speaking the text. It operates on direct input without autonomous planning or decision-making.
Limited Risk: The system is used for generating AI-generated content such as deepfakes, which under the EU AI Act requires transparency obligations to ensure users are aware they are interacting with AI-generated media.
The primary function of Synthesia is to generate deepfake videos of lifelike avatars speaking from typed text scripts, which also involves synthetic voice generation.
The reports provide clear, consistent details about the nature of the AI system, its developer (Synthesia), the actors involved (Spamouflage), and the specific videos produced. The technical mechanism of the AI (text-to-video avatar generation) and the failure of content moderation are explicitly described.
None
Technology
Highly Realistic AI Avatars
Synthesia's AI creates lifelike digital twins that mimic real humans.
Rapidly Evolving Deepfakes
Technology allows easy cloning of gestures, body language, and voices.
Generative Adversarial Networks
GANs enable the creation of unique, realistic faces not in training data.
Data Inputs
Text-to-Speech Scripts
Users typed scripts that the AI actors read out in multiple languages.
Use of Hired Actor Likenesses
Avatars are based on real actors' appearances, making them highly realistic.
Single Image Manipulation
AI can clone and manipulate faces based on just a single image input.
Human Factors
High Public Trust in Videos
People instinctively trust video content more than static images online.
Low Public Visual Literacy
General public lacks the skills to distinguish deepfakes from real videos.
Malicious Actor Intent
State-aligned groups actively exploit AI tools to spread political propaganda.
Process and Methods
Weak Moderation of Grey Areas
Moderation systems struggle to detect subtle political misinformation.
Easy Commercial Access
Low-cost subscriptions allow anyone to generate AI videos in minutes.
Ineffective Social Media Filters
Platforms like Twitter and Facebook failed to automatically block the videos.
Management
Small Content Moderation Team
Synthesia only had a four-person team dedicated to preventing misuse.
Reliance on Post-Hoc Bans
Company suspended accounts only after the deepfakes were exposed by others.
Difficult Risk Assessment
Management found it hard to identify and block grey-area misinformation.
Regulatory Environment
Lack of Clear AI Laws
Absence of comprehensive regulations to manage the spread of deepfake tools.
Lagging Regulatory Frameworks
Technological advances run far ahead of existing digital age legislation.
No Mandatory Disclosure Rules
Lack of legal requirements to label AI-generated content as synthetic.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
