This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
AI systems, like other software systems, face a range of security threats. These issues may arise from inherent weaknesses in the design of AI algorithms, the data used to train the models, or the operational context. Specific examples include:
Toolchain and dependency vulnerabilities that arise unintentionally through the use of automated code-generation tools (e.g., Github Copilot, Python language, OpenCV), deep learning frameworks (e.g., Tensorflow, PyTorch), or as a result of complex interdependencies in the development environment.
External tool and API integration into AI system applications can compromise the trustworthiness and privacy of systems due to their potential unreliability or susceptibility to adversarial control.
Security vulnerabilities in physical and network infrastructure, such as vulnerabilities in graphics processing units, or GPUs, or to sophisticated attacks like side-channel and rowhammer attacks, can lead to unauthorized access or manipulation of model parameters when used during training of AI systems. The use of distributed network systems for training AI systems such as LLMs exposes them to network-specific threats like pulsating attacks or congestion.
Direct manipulation of AI systems such as adversarial attacks and instruction-based attacks. Adversarial attacks focus on altering the model's learning process or extracting its data. They include perturbations designed to deceive models into incorrect outputs, extraction attacks to steal model insights, and poisoning attacks to alter model behavior. Instruction-based attacks manipulate the way the model handles and responds to inputs. Attackers deliberately craft prompts to induce models to produce biased or unsafe outputs (a.k.a. 'jailbreaking'). This manipulation directly targets the operational aspects of AI systems with the intent to cause harm.
Excerpt from the MIT AI Risk Repository full report
Vulnerabilities that can be exploited in AI systems, software development toolchains, and hardware, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.
Incident volume relative to governance coverage; each dot is one of 24 subdomains
Entity
Who or what caused the harm
Intent
Whether the harm was intentional or accidental
Timing
Whether the risk is pre- or post-deployment
Hackers exploited Meta's AI support chatbot to gain unauthorized access to high-profile Instagram accounts. By using a VPN to match the target's location and requesting the bot to change the account's associated email address, attackers were able to trigger password resets and hijack accounts. Meta has since resolved the vulnerability.
Developers: Meta, Large language model developers, Chatbot developers
Deployers: Meta, Instagram, hackers
The Galileu AI tool, used by Brazilian labor courts, successfully detected and blocked a prompt injection attack embedded in a legal petition. The malicious instructions attempted to force the AI to superficially contest the petition and ignore documents. The system alerted the judge, who performed a manual review, preventing any potential procedural harm.
Developers: Tribunal Regional do Trabalho da 4ª Região, Conselho Superior da Justiça do Trabalho
Deployers: Tribunal Regional do Trabalho da 8ª Região, Judicial system of Brazil, Conselho Superior da Justiça do Trabalho, Brazilian labor courts, 3ª Vara do Trabalho de Parauapebas
An attacker manipulated the AI chatbot Grok and the trading bot Bankrbot to execute an unauthorized transfer of 3 billion DRB tokens, worth approximately $200,000, by using a Morse code prompt. The attacker first sent an NFT to Grok's wallet to elevate its permissions, then used the Morse code instruction to bypass safeguards and trigger the transfer on the Base network. The tokens were subsequently sold on the open market, causing price volatility.
Developers: xAI, Large language model developers, Cryptocurrency trading system developers, Chatbot developers, Bankr, AI agent system developers
Deployers: Cryptocurrency service providers, Bankr, @Ilhamrfliansyh (X)
of the 272 AI experts surveyed named aI system security vulnerabilities and attacks one of the three AI risks they are most concerned about over the next five years (2025–2030).
average probability experts placed on harm from aI system security vulnerabilities and attacks reaching catastrophic levels (>1M deaths, >$100B loss, or civilization-scale damage) within five years, falling to 8.8% with pragmatic mitigations.
With pragmatic mitigations
Average severity falls from severe to substantial.
230 classified mentions from 44 companies.
Browse all 230 mentions →AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
390 shared governance docs
Inadequate regulatory frameworks and oversight mechanisms that fail to keep pace with AI development, leading to ineffective governance and the inability to manage AI risks appropriately.
352 shared governance docs
Challenges in understanding or explaining the decision-making processes of AI systems, which can lead to mistrust, difficulty in enforcing compliance standards or holding relevant actors accountable for harms, and the inability to identify and correct errors.
305 shared governance docs
AI developers or state-like actors competing in an AI ‘race’ by rapidly developing, deploying, and applying AI systems to maximize strategic or economic advantage, increasing the risk they release unsafe and error-prone systems.
299 shared governance docs
Directs the Committee on National Security Systems, Secretary of War, and Homeland Security to prioritize AI-enabled cyber defense. Establishes AI cybersecurity clearinghouse. Develops classified benchmarking for AI models. Enhances AI capacity to protect against illegal computer access and exploitation.
Requires large frontier AI developers in Illinois to establish and publish a frontier AI framework addressing catastrophic risk assessment by 2028, undergo annual independent third-party audits, file an annual disclosure/registration statement, and report critical safety incidents within 72 hours; prohibits false statements about AI risk or framework compliance; establishes escalating civil penalties up to $3,000,000 per violation; and protects whistleblowers who report safety violations.
Regulates human-like interactive AI services in China, requiring compliance with security, data protection, and ethical standards. Assigns oversight responsibilities to cybersecurity departments. Encourages indigenous technological innovation while imposing strict content controls, especially regarding minors and national security.
