This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A Sydney-based AI startup's text-to-image generator was found to be exploitable for creating nonconsensual sexual imagery of celebrities. Journalists discovered that users were sharing prompts on platforms like Reddit and Telegram to bypass the company's content moderation filters. The company has acknowledged the issue and is implementing additional guardrails to prevent further misuse.
Sydney-based startup Leonardo AI's text-to-image generator was alleged to have been exploited to create nonconsensual sexual images of celebrities, bypassing content moderation systems with user-shared prompts.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
The AI system generated explicit, nonconsensual pornography, exposing users on Telegram and Reddit to toxic and sexually explicit content.
The AI system's content filters were easily bypassed using simple prompt manipulation techniques, showing a lack of robustness in its safety guardrails.
The incident involves users exploiting an AI image generator to create nonconsensual sexual and humiliating deepfake imagery of specific celebrities.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was driven by human users intentionally designing and sharing prompts to bypass safety filters to generate nonconsensual pornography post-deployment.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
Users exploited a Sydney-based AI startup's text-to-image generator to create nonconsensual sexual deepfakes of celebrities. While representing a serious violation of privacy and personal dignity, the national security threat remains minor, as it does not target critical infrastructure, state sovereignty, or political institutions, and is being addressed through standard content moderation updates.
The AI system operates as a text-to-image generator, producing images directly based on user-provided text prompts without autonomous planning or independent decision-making.
Limited Risk: The system is used to generate AI-generated content such as deepfakes, which falls under Risk Level 3 due to transparency and disclosure obligations.
The AI system was exploited specifically to generate deepfake pornographic images of celebrities using text prompts.
The report clearly details the exploit, the platforms involved (Reddit, Telegram), the specific nature of the generated content (nonconsensual celebrity porn), and the response of the startup. There is minimal ambiguity regarding what occurred.
None
Technology
Weak Moderation Filters
Automated filters failed to block engineered prompts.
High-Fidelity Image Generation
The generator easily created realistic nonconsensual images.
Data Inputs
Adversarial Prompts
Users entered specific instructions to bypass existing filters.
Human Factors
Malicious User Intent
Users actively sought to generate nonconsensual sexual images.
Online Sharing of Exploits
Users shared prompt instructions on Reddit and Telegram.
Process and Methods
Inadequate Prompt Guardrails
The system lacked robust logic to address sidestepping methods.
Reactive Moderation Process
System patches were applied only after the vulnerability was reported.
Management
Rapid Platform Scaling
Rapid commercial scaling may have outpaced proactive safety measures.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
