This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Decisions made during the development of an algorithmic system and the content, quality, and diversity of the training data can significantly impact which people and experiences the system can effectively understand, represent, and accommodate. Biases and limitations introduced through these factors can lead to models that perform significantly worse for certain subpopulations compared with others, especially those defined by disability, gender identity, race, social status, and ethnicity. For example, when LLMs are trained on a small number of languages, they can underperform for others. The underperformance of algorithmic systems for certain groups may lead to a range of negative consequences such as the reduced ability or complete inability to use and benefit from the system; increased effort or challenges in using it effectively; feelings of alienation, frustration, and exclusion due to the lack of inclusive design; and ultimately, unequal outcomes across various domains.
Excerpt from the MIT AI Risk Repository full report
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Incident volume relative to governance coverage; each dot is one of 24 subdomains
Entity
Who or what caused the harm
Intent
Whether the harm was intentional or accidental
Timing
Whether the risk is pre- or post-deployment
The Washington State Department of Licensing implemented an AI-driven phone system that failed to provide requested Spanish-language support. Callers selecting the Spanish option were instead met with an AI voice speaking English with a Spanish accent. The agency attributed the error to staff configuration and technical issues, which created significant accessibility barriers for Spanish-speaking residents.
Developers: Amazon
Deployers: Washington State Department of Licensing, Amazon
UK police facial recognition technology was found to have significantly higher false positive rates for Black and Asian individuals, particularly Black women, compared to white subjects. Despite internal findings of bias, police forces lobbied to revert to a lower confidence threshold to maintain the number of investigative leads, raising concerns about the prioritization of operational convenience over civil rights and the lack of transparency regarding these system failures.
Developers: Unknown facial recognition technology developers
Deployers: Metropolitan Police, Law enforcement, Home Office, Government of the United Kingdom, British law enforcement
During an immigration enforcement operation in Oregon, ICE and CBP officers used the Mobile Fortify facial recognition app to identify a detained woman. The system returned two different and incorrect identities for the same individual. This incident highlights concerns regarding the reliability of the app, which officials have claimed provides a 'definitive' determination of immigration status, potentially leading to wrongful detention.
Developers: United States Department of Homeland Security
Deployers: United States Immigration and Customs Enforcement, United States Customs and Border Protection
of the 272 AI experts surveyed named unequal performance across groups one of the three AI risks they are most concerned about over the next five years (2025–2030).
average probability experts placed on harm from unequal performance across groups reaching catastrophic levels (>1M deaths, >$100B loss, or civilization-scale damage) within five years, falling to 5.4% with pragmatic mitigations.
With pragmatic mitigations
Average severity falls from severe to substantial.
24 classified mentions from 8 companies.
Browse all 24 mentions →Challenges in understanding or explaining the decision-making processes of AI systems, which can lead to mistrust, difficulty in enforcing compliance standards or holding relevant actors accountable for harms, and the inability to identify and correct errors.
204 shared governance docs
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
194 shared governance docs
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
191 shared governance docs
Vulnerabilities that can be exploited in AI systems, software development toolchains, and hardware, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.
155 shared governance docs
Regulates the use of automated decision-making technology (ADMT) in consequential decisions, requiring deployers to provide disclosures, facilitate consumer rights for correction and human review, and mandates record-keeping by developers. Grants the Colorado Attorney General exclusive enforcement authority, effective January 1, 2027.
Amends laws to require New York government agencies to disclose, assess, and submit reports on their use of AI decision-making tools. Ensures human oversight, prohibits biased outcomes, and protects employee rights regarding AI's impact on employment and duties.
Requires the Superintendent to convene a working group to develop AI usage guidelines for educational settings. Instructs the group to assess AI use in schools, develop model policies, and report findings by 2027. Specifies transparency, privacy, and academic integrity provisions.
