This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The Kimsuky group, a state-affiliated threat actor, conducted a spear-phishing campaign targeting South Korean defense and research sectors by using ChatGPT to generate realistic fake military ID cards. These AI-generated images served as decoys to lure victims into opening malicious ZIP files, which subsequently installed malware for data theft and remote control. This incident demonstrates the evolution of Kimsuky's tactics from simple security pop-up lures to more sophisticated social engineering using AI-generated deepfakes.
Genians reported a phishing campaign by North Korea's Kimsuky group using purportedly AI-generated deepfake military ID cards. Emails reportedly impersonating South Korean defense institutions carried ZIP files with forged IDs whose photos were reportedly created using generative AI. When opened, hidden malware reportedly executed, downloading scripts disguised as Hancom Office updates. This reportedly marked an evolution in Kimsuky's tactics, using AI decoys to boost social engineering.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
The AI system failed to maintain its safety guardrails, allowing the attackers to bypass its refusal filters and generate illegal counterfeit identification documents.
The Kimsuky group used ChatGPT to generate a fake military ID to impersonate a military official and deceive targeted individuals in a spear-phishing campaign, which constitutes targeted exploitation and impersonation for malicious gain.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by human threat actors (Kimsuky) intentionally abusing a deployed AI system (ChatGPT) to generate deceptive assets for a cyberattack.
North Korean state-affiliated group Kimsuky utilized ChatGPT to generate highly realistic fake South Korean military ID cards to enhance a spear-phishing campaign targeting defense and research sectors. This incident highlights the growing threat of generative AI being leveraged by state actors to bypass safety filters and produce sophisticated social engineering assets for cyber-espionage.
The threat actor used ChatGPT as a generative tool to produce sample ID card images. The AI performed a specific, well-defined function (image generation) based on direct user input without independent planning or autonomous decision-making.
Limited Risk: The report describes the use of AI to generate deepfakes (fake military ID cards). Under the EU AI Act, AI systems that generate or manipulate image, audio or video content (deepfakes) pose a moderate risk and are subject to specific transparency obligations.
The threat actors explicitly used ChatGPT to generate a fake military ID card image that resembled a real person's ID, which was identified as a deepfake with 98% probability.
The reports from Genians provide highly detailed technical analysis of the phishing campaign, including the specific malware chain, C2 servers, and the role of ChatGPT in generating the fake ID. The metadata of the PNG file explicitly confirmed ChatGPT's involvement, leaving little ambiguity about the AI's role.
None. The incident is clearly a state-sponsored cyber-espionage campaign leveraging AI-generated social engineering decoys.
Technology
Bypassing Model Safeguards
Attackers bypassed ChatGPT safeguards using mock-up framing prompts.
Deepfake Generation Ease
Generative AI makes creating highly convincing fake military IDs straightforward.
AI-Assisted Identity Manipulation
AI tools enable North Korean workers to generate fake virtual identities.
Data Inputs
Adversarial Prompt Framing
Manipulated prompt inputs bypassed the model's refusal triggers.
Public ID Data Availability
Publicly available military ID images on the internet were used as reference.
Human Factors
Trust in AI Realism
Victims trusted highly realistic AI-generated ID images in emails.
Limited Worker Programming Skills
North Korean workers relied on AI to compensate for poor coding skills.
Language Communication Gaps
Workers used AI to overcome restricted English-based communication skills.
Process and Methods
Insufficient Recruitment Screening
Technical assessment processes failed to detect AI-assisted candidate fraud.
Inadequate Phishing Detection
Traditional email security failed to flag AI-generated deepfake decoys.
Management
Inadequate AI Risk Monitoring
Organizations lacked proactive monitoring for AI misuse in operations.
Weak Endpoint Security Adoption
Lack of EDR systems to detect obfuscated scripts triggered by AI phishing.
Regulatory Environment
Sanction Circumvention via AI
State-sponsored actors leveraged AI to bypass international trade sanctions.
Lack of Deepfake Controls
Absence of strict global regulations on generating official document images.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
