This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A South Korean man was sentenced to two and a half years in prison for using AI to generate 360 sexually exploitative images of children. This case is noted as the first of its kind in the South Korean court system, highlighting the legal challenges posed by AI-generated abusive content. The images were confiscated by police and were not distributed.
A South Korean man used AI technology to generate 360 images of a sexual nature depicting children in April of 2023. The police confiscated the images and the courts sentenced the man to two and a half years in prison. The case marked the first of its nature in the South Korean court system.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The report also highlights cases in Spain and on Twitch where AI was used to create non-consensual sexual imagery to target, extort, or humiliate specific victims.
The incident involved the generation of child sexual abuse material (CSAM), which is explicitly classified under exposure to toxic content.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by a human user who intentionally used a deployed AI system to generate sexually exploitative images of children.
Added categories peak at 4: Severe (Child Sexual Exploitation & Abuse (CSEA), Inferred)
A South Korean court sentenced an individual to prison for generating AI-based child sexual abuse material, establishing a novel legal precedent that realistic virtual depictions of minors constitute CSAM. While representing a significant legal milestone for AI regulation and human rights protection, the incident was successfully managed by local law enforcement and poses negligible threats to broader national security.
The AI system functioned as a tool, performing the specific function of generating images based on direct user inputs without exhibiting independent planning or autonomous decision-making.
Risk Level 1: Unacceptable Risk. The system was used to generate child sexual abuse material, which represents an exploitative system targeting vulnerabilities and violating fundamental rights, posing significant dangers to society.
The AI system was used to generate highly realistic, sexually exploitative images of virtual children, which falls under deepfake image generation.
The report clearly details the legal case, the number of images generated (360), the sentence (2.5 years), and the fact that they were not distributed. While the specific AI model is not named, the nature of the incident as AI-generated CSAM is unambiguous.
None
Technology
High-Level Realistic AI Generation
AI technology creates highly realistic images of non-existent minors.
Deepfake Generation Tools
Accessible tools allow easy manipulation of images to create explicit content.
Lack of Algorithmic Safeguards
Generative models lacked built-in blocks to prevent generating explicit minors.
Data Inputs
Target Images Used Without Consent
Real photos of victims inputted into AI tools to generate explicit images.
Unrestricted Training Data
AI models trained on datasets that enable highly realistic child synthesis.
Human Factors
Malicious Intent of Users
Users deliberately leverage AI to create abusive and exploitative content.
Lack of Ethical Constraints
Perpetrators disregard consent and legal boundaries of victims.
Process and Methods
Inadequate Platform Moderation
Streaming and sharing platforms fail to proactively block deepfakes.
Delayed Policy Enforcement
Platforms update policies reactively only after major public controversies.
Management
Prioritizing Growth Over Safety
AI developers release powerful tools without adequate safety guardrails.
Regulatory Environment
Lagging AI Regulations
Governments struggle to keep pace with rapid generative AI advancements.
Ambiguous Legal Definitions
Unclear laws regarding whether virtual child imagery constitutes abuse.
Lack of Global Standards
Inconsistent enforcement and rules across different jurisdictions.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
No population impact data reported.
