This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
In the lead-up to Pakistan's 2024 general elections, malicious actors disseminated AI-generated deepfake audio and video clips falsely attributed to political leaders, including Imran Khan. These clips falsely claimed that the PTI party was boycotting the elections, aiming to mislead voters and disrupt the democratic process. Experts confirmed the audio was artificially generated, and the party had to actively debunk the misinformation. The incident highlights the growing challenge of AI-driven disinformation in electoral contexts.
A purported deepfake audio clip, falsely attributed to Imran Khan urging a PTI (Pakistan Tehreek-e-Insaf) election boycott, circulated on social media on the eve of Pakistan's general elections. This sophisticated AI-generated misinformation aimed to mislead voters, highlighting the growing challenge of digital manipulation in political discourse.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to conduct large-scale disinformation campaigns, malicious surveillance, or targeted and sophisticated automated censorship and propaganda, with the aim of manipulating political processes, public opinion, and behavior.
Also classified as
The AI-generated deepfakes spread false information regarding political candidates boycotting the election.
The deepfakes were used to conduct a coordinated disinformation campaign aimed at manipulating public opinion and the political process during Pakistan's national elections.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was driven by human actors who intentionally generated and disseminated deepfake audio and video clips post-deployment to mislead voters.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
Malicious actors deployed AI-generated deepfake audio and video of political leaders in Pakistan, falsely claiming an election boycott on the eve of the 2024 general elections. The incident represents a substantial threat to sovereignty and information security by targeting democratic processes, though it was quickly mitigated by public debunking.
The AI systems were used as tools by human actors to perform the specific function of generating synthetic audio and video (deepfakes) of political figures.
Limited Risk: The report describes the use of AI-generated content such as deepfakes, which falls under Risk Level 3 due to transparency and disclosure requirements.
The AI systems were primarily used to generate synthetic voices and deepfake videos of political figures to mimic their real likenesses.
The reports provide clear, consistent, and detailed accounts of the deepfake incidents, including expert verification of the synthetic audio and statements from the affected political party.
None
Technology
Sophisticated AI Voice Cloning
Generates highly realistic but artificial audio of political leaders.
Post-Production Manipulation
Unnatural noise was added to simulate authenticity and mask AI generation.
Lack of Real-Time Detection
Platforms lacked automated tools to instantly detect and flag AI deepfakes.
Data Inputs
Unlabeled Synthetic Audio
Deepfakes lacked metadata or watermarks indicating they were AI-generated.
Lack of Source Verification
No verification of the audio source before dissemination on social media.
Human Factors
Voter Susceptibility
Public likely to make voting decisions based on unverified false audio.
Overreliance on Social Media
Users trust and share viral content without verifying its authenticity.
Process and Methods
Delayed Fact-Checking
Manual verification processes could not keep up with viral dissemination.
Internet Shutdowns
Government internet cuts blocked journalists from debunking fake news quickly.
Ineffective Platform Moderation
Social media platforms failed to promptly remove flagged deceptive content.
Management
Growth Prioritization
Social media platforms prioritized engagement over information integrity.
Poor Incident Response
Management lacked proactive strategies to counter election-disrupting deepfakes.
Regulatory Environment
Weak Legal Frameworks
Pakistan lacked specific criminalization for malicious AI-generated content.
Inadequate Regulatory Oversight
Election and media authorities failed to enforce strict codes of conduct.
AI system involved
An AI system was involved, but the reports do not identify it.
Identified by automated extraction; can differ from the alleged developers above.
