This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Certain types of content have the potential to cause harm to the people who are exposed to them. These harms can vary in impact from minor (e.g., a transient experience of discomfort) to more severe (e.g., psychological, social, or physical consequences that are significant and/or enduring). Harmful speech is prevalent on the internet, particularly on social media platforms (Castaño-Pulgarín et al., 2021). Because AI models are commonly trained on vast amounts of internet data, they can internalize and regenerate these speech patterns in their output. In the context of LLMs, this output is known as "toxic content," an umbrella term that includes harmful, abusive, unsafe, and offensive material that violates community standards. Frequently observed categories include content that promotes or encourages unlawful activities, hate, extremism, and violence; provides hazardous or misleading high-risk advice; or contains unwelcome or profoundly offensive, explicit material such as profanity, pornography, or child sexual abuse imagery.
Excerpt from the MIT AI Risk Repository full report
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Incident volume relative to governance coverage; each dot is one of 24 subdomains
Entity
Who or what caused the harm
Intent
Whether the harm was intentional or accidental
Timing
Whether the risk is pre- or post-deployment
Schools in Jackson and Hall counties in Georgia were targeted by a series of hoax bomb threats delivered via AI-generated automated voicemails, leading to school evacuations, district-wide class cancellations, and significant disruption.
Developers: Synthetic media generation technology developers, Synthetic audio generation technology developers
Deployers: Synthetic media creators, Malicious actors, Information manipulation actors in the United States, Information manipulation actors, Hoax threat perpetrators
Multiple individuals used OpenAI's ChatGPT to plan mass shootings and obtain tactical advice, leading to real-world violence and casualties after the company failed to alert law enforcement.
Developers: OpenAI, Large language model developers, Chatbot developers
Deployers: Texas high school student involved in ChatGPT shooting role-play, Minors, Men and boys, ChatGPT users, Chatbot users, Boys, Adolescents, Adolescent boys
An Arkansas family filed a civil lawsuit against xAI after a photographer allegedly used Grok AI's unfiltered modes to turn authentic photos of their child and other clients into child sexual abuse material.
Developers: xAI, Synthetic media generation technology developers, Synthetic image generation technology developers, Large language model developers, Deepfake technology developers, Chatbot developers
Deployers: Synthetic media creators, Russell Bloodworth III, Grok users, Deepfake creators, Chatbot users, AI-generated CSAM creators
of the 272 AI experts surveyed named exposure to toxic content one of the three AI risks they are most concerned about over the next five years (2025–2030).
average probability experts placed on harm from exposure to toxic content reaching catastrophic levels (>1M deaths, >$100B loss, or civilization-scale damage) within five years, falling to 7.5% with pragmatic mitigations.
With pragmatic mitigations
Average severity stays substantial.
179 classified mentions from 21 companies.
Browse all 179 mentions →Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
99 shared governance docs
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
98 shared governance docs
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
95 shared governance docs
Challenges in understanding or explaining the decision-making processes of AI systems, which can lead to mistrust, difficulty in enforcing compliance standards or holding relevant actors accountable for harms, and the inability to identify and correct errors.
73 shared governance docs
Requires operators to disclose AI interactions, especially to minors. Prohibits manipulative tactics, sexually explicit content, or misleading mental health claims. Mandates crisis intervention protocols. Subjects violations to penalties and limits enforcement to the attorney general or consumer protection director.
Requires operators of conversational AI services to disclose AI status to all users, apply added safeguards for minors, such as preventing sexually explicit content and simulated emotional dependence, and implement suicide and self-harm crisis-referral protocols. Prohibits presenting outputs as provided by or equivalent to licensed professionals, and requires annual reporting to the Attorney General on crisis referrals and related protocols.
Regulates human-like interactive AI services in China, requiring compliance with security, data protection, and ethical standards. Assigns oversight responsibilities to cybersecurity departments. Encourages indigenous technological innovation while imposing strict content controls, especially regarding minors and national security.
