This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
In August 2024, South Korea experienced a massive surge in non-consensual sexually explicit deepfake imagery targeting women and minors. Perpetrators, often teenagers, used AI tools to manipulate social media photos of classmates, teachers, and neighbors into pornographic content, distributing them through Telegram chat groups. The incident has caused widespread panic, leading many women to delete their social media presence, and has prompted government investigations and calls for stricter regulation of both AI tools and messaging platforms.
At the end of August 2024, South Korean authorities began investigating a significant surge in the creation and dissemination, often via Telegram, of explicit deepfake pornography created without consent from the stolen social media content of female classmates, teachers, and neighbors.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
The AI tools were used to generate highly toxic, non-consensual sexually explicit content, including synthetic pornographic images of minors.
The incident involves the targeted creation of humiliating and sexually explicit synthetic imagery of specific victims (classmates, teachers, and acquaintances) for harassment and exploitation.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by human perpetrators who intentionally used deployed AI deepfake tools to generate non-consensual sexually explicit images of women and minors.
Added categories peak at 4: Severe (Child Sexual Exploitation & Abuse (CSEA), Indirect)
A massive wave of AI-generated non-consensual sexually explicit deepfakes in South Korea targeted women and minors, distributed via Telegram. The incident caused widespread societal panic, prompt government and police intervention, and highlighted significant human rights, privacy, and digital safety challenges, representing a substantial impact on societal stability.
The AI systems involved are deepfake generation apps and Telegram bots that perform specific, well-defined functions (generating a nude or swapping a face onto an explicit body) based on direct user inputs (uploading a photo) without autonomous planning or context retention.
Limited Risk: The report describes AI-generated content such as deepfakes, which under the EU AI Act are classified as limited risk, requiring specific transparency obligations to ensure users are aware the content is AI-generated.
The primary function of the AI systems used by the perpetrators was to generate sexually explicit deepfake images and videos by swapping victims' faces onto pornographic bodies.
The reports from multiple reputable sources (Bloomberg, BBC, NYT) provide consistent, detailed accounts of the deepfake wave in South Korea, including statistics from the National Police Agency and the Ministry of Education, as well as direct testimonies from victims and activists.
None
Technology
Sophisticated AI Deepfake Tools
GAN-based apps generate highly realistic explicit images in seconds.
Lack of AI Manipulation Safeguards
Insufficient technical barriers to prevent misuse of image generation tools.
Anonymity of AI Distribution Platforms
Encrypted messaging platforms allow anonymous sharing of harmful media.
Data Inputs
Unprotected Social Media Photos
Public selfies and school photos were easily harvested for deepfake models.
Lack of Image Protection Tools
Absence of active tools to prevent AI manipulation of uploaded photos.
Use of Real Faces with Fake Bodies
Combining real faces with explicit pornographic bodies to create fakes.
Human Factors
Perception of Deepfakes as a Prank
Teenagers viewed the creation of harmful deepfakes as a game or joke.
Deep-Rooted Online Misogyny
A digital culture that normalizes the sexual humiliation of women.
Lack of Digital Ethics Education
School systems focused on exams rather than teaching ethical tech use.
Process and Methods
Light-Touch Moderation Policies
Messaging apps failed to proactively moderate harmful public chatrooms.
Slow Investigative Approval Processes
Police required slow court approvals to access and shut chatrooms.
Inadequate Platform Cooperation
Platforms historically ignored police requests for investigative help.
Management
Prioritizing Profit Over Safety
Tech companies cut ethical AI teams to reduce operational costs.
Neglecting Responsible AI Rollout
Industry failed to deploy proactive safety solutions during AI rollouts.
Platform Accountability Failure
Management failed to police illegal activities occurring on their app.
Regulatory Environment
Gaps in Deepfake Legislation
No laws existed to punish buying, storing, or watching deepfake porn.
Lax Punishment for Teen Offenders
Most teen perpetrators received lenient sentences in youth courts.
Denial of Structural Sexism
Government leadership downplayed systemic gender-based discrimination.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
