This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Researchers and users have demonstrated that OpenAI's GPT-4o model can generate highly realistic, text-accurate fraudulent documents, including receipts and medical prescriptions. This capability poses a significant risk to real-world verification processes that rely on visual evidence. The report highlights that existing safety guardrails, such as watermarking, are insufficient to prevent the misuse of the model for document falsification and potential financial or medical fraud.
OpenAI's new image generator, 4o, was reportedly used to produce realistic fraudulent documents, including fake restaurant receipts, prescriptions for controlled substances, and potentially other identity or financial documents. Users demonstrated how the model could be prompted to bypass safety restrictions to be misused for expense fraud, medical forgery, and other various forms of document falsification.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
The AI system generates highly realistic but entirely fabricated documents containing false information.
The incident involves using GPT-4o to generate highly realistic fake receipts and medical prescriptions, which directly facilitates financial fraud, expense scams, and the illicit acquisition of controlled substances.
AI system
Due to a decision or action made by an AI system
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk arises post-deployment from the AI system's advanced capability to generate highly realistic text within images, which is intentionally leveraged by users to create fraudulent documents.
Added categories peak at 4: Severe (Epistemic Harm, Inferred)
Researchers demonstrated that OpenAI's GPT-4o can generate highly realistic fraudulent documents like receipts and prescriptions. While this poses a minor threat to financial, administrative, and medical verification processes, it currently represents a capability demonstration rather than an active national security crisis.
The report describes GPT-4o acting as an assistant that generates text and images based on direct user prompts (e.g., generating a fake receipt or prescription when requested by Deedy Das). It does not act autonomously or execute independent plans, but rather augments the user's capability to create realistic documents.
Risk Level 3. Limited Risk: AI systems that pose a moderate risk and require specific transparency obligations. The report describes GPT-4o as an image-generating model capable of creating realistic fake documents, which falls under AI-generated content.
The primary function of the AI system described in the incident is generating highly realistic images containing text, specifically fake receipts and prescriptions.
The reports clearly describe the capabilities demonstrated by the users and the specific documents generated (receipts, prescriptions). There is no ambiguity about the AI model involved (GPT-4o) or the nature of the risk (document fraud). While the reports discuss potential future harms rather than actual realized financial losses, this distinction is clear and easily accounted for in the assessment.
The incident could be interpreted as a security vulnerability demonstration rather than a fraud incident, but the primary focus is on the generation of fraudulent content.
Technology
Advanced Text-in-Image Generation
GPT-4o generates realistic text within images, enabling convincing forgery.
Easily Bypassed Guardrails
Metadata tags and watermarks have simple workarounds.
Human Factors
High Propensity for Fraud
Historical data shows a high rate of individuals willing to falsify receipts.
Overreliance on Visual Trust
Organizations rely on images of documents as absolute proof of validity.
Process and Methods
Weak Internal Controls
Flawed verification processes fail to detect fake documents.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
