This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
LLMs can sometimes generate content that is factually incorrect, misleading, poorly researched, or unintelligible. Risks in this category occur accidentally and not as a result of humans intentionally trying to cause harm, as is the case with disinformation.
Common sources of AI misinformation include noisy training data, sampling strategies that introduce randomness, outdated knowledge bases, and fine-tuning processes that encourage sycophantic behavior. Incorrect and misleading information generated by LLMs can result in a range of actual and anticipated negative outcomes. Individuals exposed to false information may form inaccurate beliefs and perceptions. This undermines their autonomy and ability to make free and informed choices.
Where inaccuracies in LLM predictions influence an individual's decisions and actions, the individual may experience indirect physical, emotional, or material harms especially but not exclusively in high-stakes domains such as mental health, physical health, law, and finance. For example, an LLM that offers misleading information about medical drug use may cause a consumer to harm themselves or others.
Excerpt from the MIT AI Risk Repository full report
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Incident volume relative to governance coverage; each dot is one of 24 subdomains
Entity
Who or what caused the harm
Intent
Whether the harm was intentional or accidental
Timing
Whether the risk is pre- or post-deployment
An author used AI tools ChatGPT and Claude to write a book about AI's effects on truth, but the AI generated numerous fake or misattributed quotes that were included in the published work without proper verification.
Developers: OpenAI, Anthropic
Deployers: Writers, Steven Rosenbaum, Authors
A lawyer used Anthropic's Claude AI to draft a court filing which generated false legal quotations that were submitted to a federal court, resulting in the lawyer having to apologize to the judge and implement additional safeguards.
Developers: Anthropic
Deployers: Binnall Law Group, Jason Greaves
AI image enhancement tools were used to alter real photographs of Iranian political prisoners, creating confusion between authentic documentation and AI-generated content, undermining human rights advocacy efforts.
Developers: Unknown AI Image Editing Technology Developers
Deployers: Eyal Yakoby, Iranian Embassy In South Africa, Donald Trump, White House, United States Department Of State
of the 272 AI experts surveyed named false or misleading information one of the three AI risks they are most concerned about over the next five years (2025–2030).
average probability experts placed on harm from false or misleading information reaching catastrophic levels (>1M deaths, >$100B loss, or civilization-scale damage) within five years, falling to 6.8% with pragmatic mitigations.
With pragmatic mitigations
Average severity falls from severe to substantial.
139 classified mentions from 35 companies.
Browse all 139 mentions →Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
199 shared governance docs
Challenges in understanding or explaining the decision-making processes of AI systems, which can lead to mistrust, difficulty in enforcing compliance standards or holding relevant actors accountable for harms, and the inability to identify and correct errors.
187 shared governance docs
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
178 shared governance docs
Using AI systems to conduct large-scale disinformation campaigns, malicious surveillance, or targeted and sophisticated automated censorship and propaganda, with the aim of manipulating political processes, public opinion, and behavior.
168 shared governance docs
Establishes an AI Litigation Task Force to challenge state regulations hindering United States Artificial Intelligence (AI) dominance. Directs an evaluation to identify state laws that mandate ideological bias or alter truthful model outputs. Restricts state access to federal funding, such as the Broadband Equity Access and Deployment program, unless states comply with a proposed national policy framework designed to preempt conflicting state-level AI mandates.
Defines "companion chatbot" and requires operators to notify users when they interact with AI. Requires protocols to prevent the production of harmful content. Mandates annual reports on crisis notifications. Offers civil remedies for violations. Ensures suitability disclosures for minors.
Regulates companion chatbots by requiring operators to disclose their artificial nature, implement protocols against suicide-related content, and apply heightened protections for minors. Mandates annual reporting to the Office of Suicide Prevention and subjects violators to civil liability.
