This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Researchers demonstrated that GPT-2 and GPT-3 can memorize and regurgitate verbatim training data, including personally identifiable information and copyrighted content. This vulnerability poses significant privacy and legal risks, as the models can inadvertently expose sensitive data in downstream applications. OpenAI has acknowledged these risks and is developing content-filtering systems to mitigate the unauthorized disclosure of personal information.
OpenAI's GPT-2 reportedly memorized and could regurgitate verbatim instances of training data, including personally identifiable information such as names, emails, twitter handles, and phone numbers.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
Also classified as
The model generated a false news story incorrectly attributing a real murder to an unrelated murder victim.
The models memorized and reproduced verbatim copyrighted text from books and software source code, raising copyright infringement concerns.
The incident involves language models memorizing and leaking personally identifiable information (PII) such as contact details and private chat logs from their training data.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk of data leakage is caused by the AI system's unintentional regurgitation of training data during post-deployment querying.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
Researchers demonstrated that GPT-2 and GPT-3 can memorize and regurgitate sensitive training data, including personally identifiable information and copyrighted content. This vulnerability poses minor national security risks, primarily concerning individual privacy and intellectual property protection, rather than active threats to critical infrastructure or state sovereignty.
The report describes GPT-2 and GPT-3 as language models that generate text by predicting the words that would likely follow a prompt given by a user. They augment human capabilities by generating articles, stories, or code snippets based on direct human input, fitting the Assistant profile.
Limited Risk: The report describes GPT-2 and GPT-3, which are general-purpose generative AI models and chatbots. Under the EU AI Act, chatbots and generative AI systems that interact with humans or generate synthetic content are classified as Limited Risk, requiring specific transparency obligations to notify users that they are interacting with AI.
The primary function of GPT-2 and GPT-3 is to act as a general-purpose writing assistant by generating coherent text based on user prompts. It was also tested for code generation tasks.
The reports are highly detailed, based on peer-reviewed academic research ('Extracting Training Data from Large Language Models') and technical journalism. They provide specific statistics, methodology, and concrete examples of the memorized data (PII, copyrighted text, and false attributions). There is no conflicting information regarding the core findings.
None. The technical mechanism of memorization and regurgitation is well-established and agreed upon by both the researchers and OpenAI.
Technology
Model Memorization Capacity
Large language models memorize and regurgitate rare training data.
Lack of Output Filtering
Initial models lacked systems to block or rewrite sensitive outputs.
Differential Privacy Limits
Standard DP is ineffective for data repeated across multiple sites.
Data Inputs
Uncurated Web Scraping
Scraped training datasets contain leaked PII and copyrighted text.
Lack of Data Sanitization
No effective automated process to clean sensitive data from corpus.
Inclusion of Copyrighted Works
Books and source code were included in the training dataset.
Human Factors
Adversarial Prompting
Users can craft specific queries to extract memorized training data.
Lack of Output Curation
Deploying raw model outputs directly to users without human review.
Context Misalignment
Using web-scraped data outside of its originally intended context.
Process and Methods
Flawed Dataset Curation
Relying on broad web scrapes instead of carefully curated datasets.
Inadequate Testing for Leakage
Lack of rigorous testing to identify memorized data before release.
Lack of Robust Post-Processing
Absence of real-time filters to detect and replace sensitive data.
Management
Commercialization Pressure
Prioritizing rapid API commercialization over comprehensive safety.
Underfunded Data Collection
Investing heavily in model scaling while underfunding data curation.
Inadequate Risk Assessment
Underestimating the feasibility of training data extraction attacks.
Regulatory Environment
Unclear Data Deletion Laws
Legal gray area regarding right to be forgotten in trained weights.
Ambiguous Fair Use Standards
Lack of clear legal boundaries for training models on copyrighted data.
Inadequate Privacy Protections
Existing frameworks like GDPR do not easily map to neural networks.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
