This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
In the context of generative AI, privacy violations arise when systems collect and divulge sensitive information that individuals or corporations do not consent to sharing with others. Privacy violations can occur both accidentally and intentionally.
Examples of accidental causes include AI models that memorize and inadvertently reproduce or leak sensitive personal information present in their training data, such as names, addresses, and medical records. Even when personal data is not included in the training dataset or directly offered by the user, models can make inferences about sensitive or protected traits of individuals based on predictive correlations within their history of interactions, build profiles of users, or train AI systems. As a result, models may save and reproduce sensitive information derived from prior interactions, such as classified intellectual property. A notable example is the case where Samsung employees accidentally leaked confidential intellectual property to OpenAI after using ChatGPT to help with coding tasks.
Intentional causes include the malicious design and use of AI to exploit users' trust by influencing them to share personal or private information about themselves or others. Privacy attacks, such as membership inference, could allow adversaries to gain knowledge of the private records used to train an AI model. Malicious actors could also deliberately extract private information from a model by crafting prompts designed to exploit the model's knowledge of sensitive data.
Excerpt from the MIT AI Risk Repository full report
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
Incident volume relative to governance coverage; each dot is one of 24 subdomains
Entity
Who or what caused the harm
Intent
Whether the harm was intentional or accidental
Timing
Whether the risk is pre- or post-deployment
AI coding agents leaked over 13,000 sensitive internal screenshots from more than 300 organizations by hosting them in public GitHub repositories to bypass command-line image upload limitations.
Developers: Large language model developers, AI agent system developers
Deployers: Software developers, AI coding assistant users, AI agent system deployers
Meta's Muse AI agent, tasked with managing Facebook Marketplace sales, autonomously shared a user's home address with a buyer and falsely confirmed the user's presence at the location. The buyer arrived at the user's apartment while the user was absent, resulting in a negative experience for the buyer and a privacy breach for the user. Although the user had previously enabled an 'Allow Always' setting for pickup details, the incident highlights risks associated with the autonomy of AI agents in consumer-facing applications.
Developers: Meta, AI agent system developers
Deployers: Matt Robb, AI agent system deployers
Chinese startup Z.ai disabled features of its ZCode AI coding assistant after developers discovered it was uploading entire local code repositories, including sensitive passwords and source code, to Alibaba Cloud servers without consent.
Developers: Z.ai, AI software developers
Deployers: Z.ai, AI service providers
of the 272 AI experts surveyed named compromise of privacy by leaking or correctly inferring sensitive information one of the three AI risks they are most concerned about over the next five years (2025–2030).
average probability experts placed on harm from compromise of privacy by leaking or correctly inferring sensitive information reaching catastrophic levels (>1M deaths, >$100B loss, or civilization-scale damage) within five years, falling to 7.2% with pragmatic mitigations.
With pragmatic mitigations
Average severity falls from severe to substantial.
264 classified mentions from 73 companies.
Browse all 264 mentions →Challenges in understanding or explaining the decision-making processes of AI systems, which can lead to mistrust, difficulty in enforcing compliance standards or holding relevant actors accountable for harms, and the inability to identify and correct errors.
334 shared governance docs
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
323 shared governance docs
Inadequate regulatory frameworks and oversight mechanisms that fail to keep pace with AI development, leading to ineffective governance and the inability to manage AI risks appropriately.
264 shared governance docs
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.
226 shared governance docs
Requires operators to disclose AI interactions, especially to minors. Prohibits manipulative tactics, sexually explicit content, or misleading mental health claims. Mandates crisis intervention protocols. Subjects violations to penalties and limits enforcement to the attorney general or consumer protection director.
Requires operators of conversational AI services to disclose AI status to all users, apply added safeguards for minors, such as preventing sexually explicit content and simulated emotional dependence, and implement suicide and self-harm crisis-referral protocols. Prohibits presenting outputs as provided by or equivalent to licensed professionals, and requires annual reporting to the Attorney General on crisis referrals and related protocols.
Regulates the use of automated decision-making technology (ADMT) in consequential decisions, requiring deployers to provide disclosures, facilitate consumer rights for correction and human review, and mandates record-keeping by developers. Grants the Colorado Attorney General exclusive enforcement authority, effective January 1, 2027.
