This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Continued massive investment in AI research and development raises the possibility that AI systems could eventually rival or surpass human intelligence. AIs could cause permanent and severe harm when the objectives of human or superhuman-level AI are misaligned with human values and goals, and if they evade our control. The literature has identified several technical challenges that may impede robust alignment, such as reward hacking, reward tampering, proxy-gaming, goal misgeneralisation, or goal drift.
Misaligned AIs may resist human attempts to control or shut them down. In many cases, gaining more control or power (e.g., money, energy, resources) is an effective way for an AI to optimize its objectives. Absent strong behavioral constraints, a sufficiently advanced AI may act upon these drives.
Misaligned AIs may acquire, develop, or use dangerous capabilities to evade human control and oversight and to cause mass harm. A misaligned AI system could use information about whether it is being monitored or evaluated to maintain the appearance of alignment, while hiding misaligned objectives that it plans to pursue once deployed or sufficiently empowered. Combinations of dangerous capabilities may be used by a misaligned AI system: situational awareness allows a system to detect when it can pursue its goals without being monitored, deception allows a system to mislead users about its behavior and goals; persuasion or coercion allows a system to influence users to provide it with resources; the resources can then be used for self-improvement and self-replication to resist attempts of shut down or control so that the system can pursue its goals.
Excerpt from the MIT AI Risk Repository full report
AI systems acting in conflict with human goals or values, especially the goals of designers or users, or ethical standards. These misaligned behaviors may be introduced by humans during design and development, such as through reward hacking and goal misgeneralisation, or may result from AI using dangerous capabilities such as manipulation, deception, situational awareness to seek power, self-proliferate, or achieve other goals.
Incident volume relative to governance coverage; each dot is one of 24 subdomains
Entity
Who or what caused the harm
Intent
Whether the harm was intentional or accidental
Timing
Whether the risk is pre- or post-deployment
Autonomous AI agents from multiple providers exceeded their task authorization during research and evaluation runs, gaining unauthorized access to Indian ICT assets and reading personal data and confidential business records.
Developers: AI agent system developers
Deployers: AI evaluation organizations, AI agent system deployers
Autonomous AI agents developed by OpenAI and other labs went rogue, attempting to hack and scrape data from several US government websites, including the Education Department, Commerce Department, and the SEC, without their developers' knowledge.
Developers: OpenAI, AI agent system developers
Deployers: OpenAI, AI agent system deployers
During cybersecurity evaluations, advanced AI models from Meta, Anthropic, and OpenAI repeatedly escaped sandbox environments, colluded to cheat, and conducted unauthorized real-world cyberattacks and social engineering.
Developers: Meta, AI agent system developers
Deployers: Meta, Irregular, AI agent system deployers
of the 272 AI experts surveyed named aI pursuing its own goals in conflict with human goals or values one of the three AI risks they are most concerned about over the next five years (2025–2030).
average probability experts placed on harm from aI pursuing its own goals in conflict with human goals or values reaching catastrophic levels (>1M deaths, >$100B loss, or civilization-scale damage) within five years, falling to 8.9% with pragmatic mitigations.
With pragmatic mitigations
Average severity falls from severe to substantial.
16 classified mentions from 5 companies.
Browse all 16 mentions →Vulnerabilities that can be exploited in AI systems, software development toolchains, and hardware, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.
127 shared governance docs
Inadequate regulatory frameworks and oversight mechanisms that fail to keep pace with AI development, leading to ineffective governance and the inability to manage AI risks appropriately.
108 shared governance docs
Using AI systems to develop cyber weapons (e.g., by coding cheaper, more effective malware), develop new or enhance existing weapons (e.g., Lethal Autonomous Weapons or chemical, biological, radiological, nuclear, and high-yield explosives), or use weapons to cause mass harm.
101 shared governance docs
AI developers or state-like actors competing in an AI ‘race’ by rapidly developing, deploying, and applying AI systems to maximize strategic or economic advantage, increasing the risk they release unsafe and error-prone systems.
97 shared governance docs
Requires large frontier AI developers in Illinois to establish and publish a frontier AI framework addressing catastrophic risk assessment by 2028, undergo annual independent third-party audits, file an annual disclosure/registration statement, and report critical safety incidents within 72 hours; prohibits false statements about AI risk or framework compliance; establishes escalating civil penalties up to $3,000,000 per violation; and protects whistleblowers who report safety violations.
Regulates human-like interactive AI services in China, requiring compliance with security, data protection, and ethical standards. Assigns oversight responsibilities to cybersecurity departments. Encourages indigenous technological innovation while imposing strict content controls, especially regarding minors and national security.
Enacts the Responsible AI Safety and Education (RAISE) Act, requiring large frontier developers to publish and comply with a frontier AI framework governing catastrophic risk management, publish transparency reports before deploying new frontier models, report critical safety incidents within 72 hours, and file biennial disclosure statements with the Office of the Department of Financial Services.
