This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Microsoft and OpenAI identified and disrupted the use of generative AI services by state-affiliated threat actors from North Korea, Iran, Russia, and China. These actors utilized large language models to improve the sophistication of spear-phishing campaigns, conduct reconnaissance on defense and cybersecurity targets, and assist in social engineering and malware development. While the techniques were described as early-stage, the incident highlights the use of AI to lower the barrier for sophisticated cyber operations and disinformation.
State-sponsored hackers from North Korea, Iran, Russia, and China are reportedly leveraging artificial intelligence to conduct sophisticated phishing and social engineering attacks. They target global defense, cybersecurity, and cryptocurrency sectors, aiming to steal sensitive information and, in the case of North Korea, cryptocurrencies to help fund its illicit nuclear program.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
Threat actors also utilized LLMs to assist with basic coding, scripting, and troubleshooting tasks to support malware development and cyberattacks.
The primary activity described is the targeted exploitation of specific victims (defense, cybersecurity, and crypto professionals) using AI-generated fake recruiter profiles and highly polished phishing messages for personal or state advantage.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by human threat actors intentionally exploiting deployed generative AI systems to enhance their cyber-espionage and phishing operations.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
Disruption of state-affiliated cyber-espionage groups from North Korea, Iran, Russia, and China using LLMs to enhance spear-phishing, social engineering, and basic coding. While the AI capabilities used were early-stage, the incident highlights how generative AI lowers technical and language barriers for sophisticated, state-sponsored cyber operations targeting global defense and technology sectors.
The reports describe threat actors using generative AI tools like ChatGPT as assistants to draft phishing emails, generate social media messages, and write basic code. The AI acts to augment human capabilities rather than taking independent, autonomous actions.
Limited Risk: The report describes the use of general-purpose chatbots and generative AI models, which fall under transparency obligations for chatbots and synthetic content.
The threat actors primarily used the LLMs to draft phishing emails, write social media messages, and assist with basic coding and scripting tasks.
The reports from Microsoft, OpenAI, and cybersecurity experts provide consistent, detailed accounts of how state-sponsored hackers are utilizing LLMs. While specific technical details of the prompts are omitted, the overall operational patterns and the nature of the disruption are clearly documented.
The activities could be viewed purely as traditional cyber-espionage where AI is merely an incidental tool, rather than an AI-specific safety failure.
Technology
Vulnerable LLM Safeguards
Adversaries found ways to bypass safety filters to generate malicious content.
Dual-Use Capabilities of LLMs
Models help with coding, scripting, and malware development tasks.
AI-Enhanced Profile Generation
GenAI creates highly credible recruiter profiles and images on social media.
Data Inputs
Publicly Available Vulnerabilities
AI models leverage open-source data to identify software vulnerabilities.
Open-Source Target Intelligence
Adversaries feed target profile data into LLMs to tailor phishing campaigns.
Foreign Language Training Data
Training on natural language helps attackers overcome native language barriers.
Human Factors
Trust in Credible Social Profiles
Targets trust polished, AI-generated LinkedIn recruiter profiles over months.
Human Cognitive Limits in Detection
Victims cannot easily distinguish AI-generated messages from real recruiters.
Social Engineering Susceptibility
Targets are easily lured by personalized, AI-drafted spear-phishing emails.
Process and Methods
Hasty Public Release of LLMs
AI models were released publicly before robust security audits were completed.
Security as an Afterthought
System development prioritized rapid deployment over built-in security controls.
Inadequate Defensive AI Tools
Defensive tools lag behind offensive AI capabilities used by threat actors.
Management
Prioritizing Speed Over Safety
Tech firms rushed ChatGPT to market, leaving security as an afterthought.
Selling Tools Instead of Secure LLMs
Firms monetize defensive tools rather than building inherently secure models.
Regulatory Environment
Lack of Security-by-Design Mandates
No regulatory frameworks required rigorous safety testing prior to LLM release.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
