This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A viral AI-generated image of Pope Francis wearing a white puffer jacket was created using Midjourney v5 and widely circulated on social media. The image successfully deceived millions of viewers, leading to significant public confusion. The incident prompted the Pope to issue warnings regarding the dangers of AI-generated disinformation and the distortion of reality.
A viral image of Pope Francis wearing a white puffer jacket was a deepfake produced by the photorealistic-image-generator Midjourney.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The incident highlights the broader risk of synthetic media reproducing and imitating public likenesses, which the Pope warned could affect the foundations of information and journalism.
The incident involved the generation and viral spread of a discrete false image that misled millions of people into believing the Pope wore a designer jacket.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk of widespread public deception occurred post-deployment due to the AI system generating an unexpectedly realistic image of a public figure that fooled millions.
Added categories peak at 4: Severe (Epistemic Harm, Direct)
An individual used Midjourney v5 to generate a photorealistic deepfake of Pope Francis in a puffer jacket, which went viral and deceived millions. While illustrating the growing sophistication of synthetic media and epistemic challenges, the incident carries negligible direct national security implications as it was a benign, non-state-sponsored event with no physical or structural damage.
The Midjourney software operated as a tool, performing the specific function of generating an image based on direct text prompts provided by a user without autonomous planning or context retention.
Limited Risk: The incident involves an AI system generating synthetic content (a deepfake), which under the EU AI Act is subject to specific transparency obligations to ensure users are informed that the content is AI-generated.
The AI system was used specifically to generate a highly realistic, synthetic image of a real person (Pope Francis) to depict him in a scenario that never occurred.
The reports provide consistent, detailed accounts of the creation, virality, and public reception of the Pope deepfake image, including direct quotes from the Pope himself acknowledging the incident.
None
Technology
Advanced Image Synthesis
Midjourney v5 generated photorealistic images that bypassed human detection.
Unreliable Watermarking
Lack of robust watermarking technology allowed fakes to pass as authentic.
No Specialist Training Required
Generative tools allowed anyone to produce realistic images with text prompts.
Data Inputs
Lack of Origin Metadata
Metadata-tagging projects were not widely supported to verify image sources.
Human Factors
Cognitive Biases of Users
Many online users believed the fake image without verifying its authenticity.
Inability to Detect Artifacts
Subtle visual flaws like hand distortions were overlooked by the public.
Excitement and disorientation
Rapid spread of innovations outpaced public understanding of synthetic media.
Process and Methods
Rapid Social Media Virality
Platforms lacked mechanisms to halt the rapid spread of unverified fakes.
Insufficient Authentication
No standard verification processes existed before images went viral.
Management
Unrestricted Tool Release
AI developers released powerful generation tools without robust safeguards.
Priority of Innovation over Safety
Rapid deployment of synthetic tools outpaced defensive detection methods.
Regulatory Environment
Absence of Binding Treaties
There was a lack of legally binding international treaties to regulate AI.
Lack of Mandatory Labeling
No regulatory mandates existed to force platforms to label synthetic content.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
