This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
During the 2024 Indian general elections, political parties and consultants have deployed AI-generated voice clones and video avatars to conduct personalized voter outreach at scale. These campaigns often lack disclosure, leading voters to believe they are interacting with real candidates. While proponents argue this improves engagement, critics highlight the risks of deception, lack of consent, and the potential for misinformation in a democratic process.
During the 2024 Indian elections, politicians used AI-generated deepfakes to reach voters, who might be unaware they're interacting with digital clones. Providers like Divyendra Singh Jadoun of Polymath Synthetic Media Solutions created deepfakes for personalized messages. This practice, used by various political parties, is not truthful, as voters may be misled by AI-generated content posing as genuine interactions with political figures.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to conduct large-scale disinformation campaigns, malicious surveillance, or targeted and sophisticated automated censorship and propaganda, with the aim of manipulating political processes, public opinion, and behavior.
Also classified as
The AI voice clones generate synthetic speech that misleads voters into believing they are having real, personalized conversations with political candidates.
The incident involves the coordinated, large-scale deployment of AI-generated voice clones and video avatars to influence voters and manipulate political processes during an election.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The deployment of AI voice clones and video avatars was an intentional decision by human political campaigns and consultants to influence voters during the post-deployment phase of the election.
Added categories peak at 4: Severe (Epistemic Harm, Direct)
During the 2024 Indian elections, political parties deployed over 50 million undisclosed AI voice clones and video avatars for personalized voter outreach. While not a kinetic threat, the unprecedented scale of synthetic media targeting democratic processes represents a substantial information warfare and societal stability concern that could undermine public trust in democratic institutions.
The report describes interactive 'AI agent calls' that function as a multi-step pipeline where speech-to-text, the Mistral large language model, and text-to-speech voice cloning coordinate to conduct back-and-forth conversations with voters.
Limited Risk: The incident involves chatbots and AI-generated content such as deepfakes, which under the EU AI Act are subject to transparency obligations to ensure users are aware they are interacting with AI.
The primary application of the AI systems described is cloning the voices of politicians to make millions of automated outreach calls, alongside generating synthetic video avatars of candidates.
The report provides extensive, detailed, and first-hand journalistic accounts of the deployment of AI voice clones and deepfakes in the 2024 Indian elections, including interviews with developers, consultants, and politicians. The scale of the deployment (over 50 million calls) and the specific technologies used (DeepFaceLab, Mistral) are clearly documented.
Some might interpret these AI campaigns as benign voter engagement and translation tools rather than malicious political manipulation, though the lack of disclosure strongly points to epistemic and democratic risks.
Technology
AI Agent Hallucinations
LLM generated incorrect replies, loops, and screaming audio responses.
High Latency in AI Agent Calls
Delays in machine responses disrupted natural back-and-forth conversation.
Flawed Voice Synthesis
Cloned voices had anglicized pronunciations and unnatural tones.
Data Inputs
Lack of Voice Data Consent
Using speeches of deceased politicians without family consent to clone voices.
Misuse of Target Voter Data
Political apps collect and exploit personal data for hyper-targeted AI calls.
Human Factors
Voter Deception Vulnerability
Less tech-savvy rural voters fail to realize they are speaking to AI clones.
Lack of AI Literacy
Politician candidates and voters do not understand how AI cloning works.
Process and Methods
Absence of AI Disclosure
Campaigns deploy deepfakes without labeling them as AI-generated content.
Inadequate Testing of AI Agents
Deploying live interactive AI agents to voters without proper quality checks.
Management
Prioritizing Scale Over Safety
Firms and politicians prioritize massive outreach over AI safety and consent.
Regulatory Environment
Lack of Deepfake Regulations
Weak regulatory oversight allows unchecked deployment of political voice clones.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
