This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Governance failure refers to the risks and harms that arise when institutional, regulatory, and policy mechanisms fall short of effectively managing and overseeing the development and deployment of AI systems. Several issues make robust AI governance challenging to implement.
First, it is difficult to determine who is responsible or liable when AI systems fail or make decisions that result in negative consequences. At present, there exists no comprehensive framework specifically designed to assign legal responsibility to AI agents. Traditional legal principles are based on human actors, whose intentions and actions can generally be identified and judged. AI's decision-making, on the other hand, is often unpredictable, opaque, and involves complex interactions between millions of parameters. In the absence of a regulatory or legal incentive to take safety engineering seriously, developers may release poorly designed AI systems, and people harmed by those systems may be left without recourse.
A second challenge for effective AI governance is the rapid pace at which AI systems evolve. Typical governance and policy processes are inherently slow. The mismatch between the speed of AI advancements and their regulation may result in immature regulations that overlook important aspects of AI governance.
A third challenge for effective governance is an inability to influence AI developers and deployers to take safe actions. Frequently, this inability is driven by an asymmetry of information between technology companies and regulators. Technology companies often have far better knowledge about the capabilities, functioning, and potential uses of their AI systems. Without access to this knowledge, regulators can find it difficult to craft targeted rules that address the specific challenges posed by AI.
Excerpt from the MIT AI Risk Repository full report
Inadequate regulatory frameworks and oversight mechanisms that fail to keep pace with AI development, leading to ineffective governance and the inability to manage AI risks appropriately.
Incident volume relative to governance coverage; each dot is one of 24 subdomains
Entity
Who or what caused the harm
Intent
Whether the harm was intentional or accidental
Timing
Whether the risk is pre- or post-deployment
Axon Enterprise proposed a project to develop remotely operated, taser-armed drones for use in schools to combat mass shootings. The proposal was made without the approval of the company's AI ethics board, which had previously expressed concerns about weaponizing drones. The announcement led to the mass resignation of the board members, who criticized the idea as dangerous and impractical, prompting Axon to pause the project.
Developers: Axon Enterprise
Deployers: none
of the 272 AI experts surveyed named governance failure one of the three AI risks they are most concerned about over the next five years (2025–2030).
average probability experts placed on harm from governance failure reaching catastrophic levels (>1M deaths, >$100B loss, or civilization-scale damage) within five years, falling to 7.3% with pragmatic mitigations.
With pragmatic mitigations
Average severity falls from severe to substantial.
29 classified mentions from 17 companies.
Browse all 29 mentions →Vulnerabilities that can be exploited in AI systems, software development toolchains, and hardware, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.
352 shared governance docs
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
307 shared governance docs
Challenges in understanding or explaining the decision-making processes of AI systems, which can lead to mistrust, difficulty in enforcing compliance standards or holding relevant actors accountable for harms, and the inability to identify and correct errors.
298 shared governance docs
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
264 shared governance docs
Requires large frontier AI developers in Illinois to establish and publish a frontier AI framework addressing catastrophic risk assessment by 2028, undergo annual independent third-party audits, file an annual disclosure/registration statement, and report critical safety incidents within 72 hours; prohibits false statements about AI risk or framework compliance; establishes escalating civil penalties up to $3,000,000 per violation; and protects whistleblowers who report safety violations.
Enacts the Responsible AI Safety and Education (RAISE) Act, requiring large frontier developers to publish and comply with a frontier AI framework governing catastrophic risk management, publish transparency reports before deploying new frontier models, report critical safety incidents within 72 hours, and file biennial disclosure statements with the Office of the Department of Financial Services.
Establishes a framework for managing catastrophic AI risks, recommending industry-wide safety measures. Requires Risk Reports and Frontier Safety Roadmaps to inform AI development. Outlines governance commitments, including the role of the Responsible Scaling Officer and procedures for internal and external reviews.
