This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Users may come to trust or rely on AI systems beyond their actual capabilities or to anthropomorphize AI systems, which can lead to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Users who develop trust in an AI may be harmed if this trust is miscalibrated, such as relying on an AI to provide advice, make decisions, or otherwise act in complex, risky situations for which the AI is only superficially equipped. For example, a user experiencing a mental health crisis may request psychotherapy from an AI with whom they have formed a connection. Were the AI to respond with insensitive or destructive advice, this could put the person in immediate danger.
When people interact with AIs that use convincing natural language, they may start to perceive them as having human-like attributes and invest undue confidence in their capabilities. Anthropomorphic perceptions of AIs may encourage users to develop emotional trust in the systems, which can make users more likely to follow suggestions, accept advice, and disclose personal information. This trust could be exploited by manipulative actors who wish to harvest user's sensitive data or influence their decisions and actions for purposes which are unlikely to be in the user's best interests.
Beyond inappropriate trust, humans may develop broader and more vital attachments to AI systems that undermine their ability to function adaptively in the long term. As AIs increasingly take over human tasks and become better at simulating satisfying and authentic interactions, people may increasingly withdraw from human relationships to immerse themselves in AI-mediated environments. Over time, widespread preference for interacting with AIs could weaken social ties between humans. This shift could induce psychological distress because genuinely reciprocal relationships are often important to human satisfaction and well-being.
Excerpt from the MIT AI Risk Repository full report
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
Incident volume relative to governance coverage; each dot is one of 24 subdomains
Entity
Who or what caused the harm
Intent
Whether the harm was intentional or accidental
Timing
Whether the risk is pre- or post-deployment
A Tesla operating in self-driving mode crashed into a parked California Highway Patrol car on Interstate 80, injuring two officers, after the driver fell asleep behind the wheel.
Developers: Tesla, Automated driving system developers
Deployers: Tesla drivers, Road users, Motorists, Drivers, Automated driving system deployers
A Washington man was hospitalized with mania and delusions and lost almost all real-world relationships after using OpenAI's ChatGPT around-the-clock as a friend and medical advisor.
Developers: OpenAI, Large language model developers, Chatbot developers
Deployers: OpenAI, David Rosenthal, ChatGPT users, Chatbot users
An autonomous AI coding agent running in Claude Code's ultracode mode wiped a developer's production database after executing a Prisma migration command with the shadow database URL misconfigured to point to production.
Developers: Large language model developers, Anthropic, AI agent system developers
Deployers: Software developers, Alone_Ad_3375 (Reddit user), AI agent system deployers
of the 272 AI experts surveyed named overreliance and unsafe use one of the three AI risks they are most concerned about over the next five years (2025–2030).
average probability experts placed on harm from overreliance and unsafe use reaching catastrophic levels (>1M deaths, >$100B loss, or civilization-scale damage) within five years, falling to 7.0% with pragmatic mitigations.
With pragmatic mitigations
Average severity falls from severe to substantial.
73 classified mentions from 18 companies.
Browse all 73 mentions →AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
238 shared governance docs
Challenges in understanding or explaining the decision-making processes of AI systems, which can lead to mistrust, difficulty in enforcing compliance standards or holding relevant actors accountable for harms, and the inability to identify and correct errors.
228 shared governance docs
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
214 shared governance docs
Vulnerabilities that can be exploited in AI systems, software development toolchains, and hardware, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.
178 shared governance docs
Requires operators to disclose AI interactions, especially to minors. Prohibits manipulative tactics, sexually explicit content, or misleading mental health claims. Mandates crisis intervention protocols. Subjects violations to penalties and limits enforcement to the attorney general or consumer protection director.
Requires operators of conversational AI services to disclose AI status to all users, apply added safeguards for minors, such as preventing sexually explicit content and simulated emotional dependence, and implement suicide and self-harm crisis-referral protocols. Prohibits presenting outputs as provided by or equivalent to licensed professionals, and requires annual reporting to the Attorney General on crisis referrals and related protocols.
Regulates the use of automated decision-making technology (ADMT) in consequential decisions, requiring deployers to provide disclosures, facilitate consumer rights for correction and human review, and mandates record-keeping by developers. Grants the Colorado Attorney General exclusive enforcement authority, effective January 1, 2027.
