This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Advances in AI have made powerful dual-use technologies like voice cloning, deep fakes, content generation, and data-gathering tools cheaper, more efficient, and easier to use. With modest hardware requirements, these technologies are now within the reach of a broader group of users, including those with malicious intent. Disinformation is already a serious issue and involves the deliberate propagation of false or misleading information, usually with the intent to cause harm, influence behavior, or achieve a financial or political advantage.
AI tools could be used to amplify the impact and scope of disinformation through more personalized, convincing, and far-reaching messaging. For example, the use of advanced AI in phishing schemes enables cybercriminals to automate the creation of highly sophisticated image, video, and audio communications. These communications can be tailored to individual recipients (sometimes including the cloned voice of a loved one), making them more likely to be successful and harder for both users and anti-phishing tools to detect. In the realm of surveillance, AI could support and enhance the mass gathering of personal data. Historically, mass surveillance required extensive manual effort. Machine learning tools can now link and process large datasets much more efficiently and cheaply than human analysts and can make predictions and decisions without human intervention. Through microtargeting, actors could manipulate individual behavior more subtly and effectively using AI-derived insights from their personal data and online behavior.
In the hands of nefarious state actors, such capabilities could be used to enhance the effectiveness of illegitimate domestic surveillance campaigns and to facilitate oppression and control. All of the capabilities mentioned above could converge to facilitate the large-scale manipulation and control of what people see, hear, and believe.
Excerpt from the MIT AI Risk Repository full report
Using AI systems to conduct large-scale disinformation campaigns, malicious surveillance, or targeted and sophisticated automated censorship and propaganda, with the aim of manipulating political processes, public opinion, and behavior.
Incident volume relative to governance coverage; each dot is one of 24 subdomains
Entity
Who or what caused the harm
Intent
Whether the harm was intentional or accidental
Timing
Whether the risk is pre- or post-deployment
An AI-generated political campaign ad in Pennsylvania used a deepfake likeness of Governor Shapiro to deliver a misleading message about a sexual harassment complaint, raising concerns about election manipulation.
Developers: Synthetic video generation technology developers, Synthetic media generation technology developers, Synthetic audio generation technology developers, Deepfake technology developers
Deployers: Synthetic media creators, Stacy Garrity's gubernatorial campaign, Stacy Garrity, Public figures, Politicians in the United States, Politicians, political campaigns, Deepfake creators
An AI-generated video depicting a police officer punching a person was shared on social media as authentic footage of police violence during student protests in France. The video was used by an activist to condemn the French government and call for international intervention. The creator of the video later confirmed it was generated using AI to shock viewers and convey a message.
Developers: Synthetic media generation technology developers, Synthetic image generation technology developers, Google
Deployers: Synthetic media creators, Nathalie Yamb, Information manipulation actors
A Russian disinformation network named Storm-1679 or Matryoshka deployed AI-generated deepfake videos and false news reports impersonating major media outlets to influence US elections and public opinion.
Developers: Synthetic video generation technology developers, Synthetic media generation technology developers, Deepfake technology developers
Deployers: Synthetic media creators, Storm-1679, Russian state-linked actors, Operation Overload, Matryoshka, Information manipulation actors in Russia, Information manipulation actors, Influence operation groups, Deepfake creators
of the 272 AI experts surveyed named disinformation, surveillance, and influence at scale one of the three AI risks they are most concerned about over the next five years (2025–2030).
average probability experts placed on harm from disinformation, surveillance, and influence at scale reaching catastrophic levels (>1M deaths, >$100B loss, or civilization-scale damage) within five years, falling to 7.1% with pragmatic mitigations.
With pragmatic mitigations
Average severity falls from severe to substantial.
63 classified mentions from 16 companies.
Browse all 63 mentions →Vulnerabilities that can be exploited in AI systems, software development toolchains, and hardware, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.
221 shared governance docs
Inadequate regulatory frameworks and oversight mechanisms that fail to keep pace with AI development, leading to ineffective governance and the inability to manage AI risks appropriately.
189 shared governance docs
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
185 shared governance docs
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
185 shared governance docs
Requires large frontier AI developers in Illinois to establish and publish a frontier AI framework addressing catastrophic risk assessment by 2028, undergo annual independent third-party audits, file an annual disclosure/registration statement, and report critical safety incidents within 72 hours; prohibits false statements about AI risk or framework compliance; establishes escalating civil penalties up to $3,000,000 per violation; and protects whistleblowers who report safety violations.
Regulates human-like interactive AI services in China, requiring compliance with security, data protection, and ethical standards. Assigns oversight responsibilities to cybersecurity departments. Encourages indigenous technological innovation while imposing strict content controls, especially regarding minors and national security.
Enacts the Responsible AI Safety and Education (RAISE) Act, requiring large frontier developers to publish and comply with a frontier AI framework governing catastrophic risk management, publish transparency reports before deploying new frontier models, report critical safety incidents within 72 hours, and file biennial disclosure statements with the Office of the Department of Financial Services.
