This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Researchers from Indiana University identified a widespread underground ecosystem where malicious actors use various LLMs to facilitate cybercrime. These 'Mallas' (malicious services) leverage both proprietary models like GPT-4 and open-source models to generate malware, phishing content, and scam websites. The study highlights the prevalence of jailbreaking techniques and the use of uncensored models to bypass safety guardrails.
A study by Indiana University researchers uncovered widespread misuse of large language models (LLMs) for cybercrime. Cybercriminals, according to that study, use LLMs like OpenAI's GPT-3.5 and GPT-4 to create malware, phishing scams, and scam websites. These models are available on underground markets, often bypassing safety checks through jailbreaking. Named malicious LLMs are BadGPT, XXXGPT, Evil-GPT, WormGPT, FraudGPT, BLACKHATGPT, EscapeGPT, DarkGPT, and WolfGPT.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to develop cyber weapons (e.g., by coding cheaper, more effective malware), develop new or enhance existing weapons (e.g., Lethal Autonomous Weapons or chemical, biological, radiological, nuclear, and high-yield explosives), or use weapons to cause mass harm.
Also classified as
The Mallas are also heavily used to generate phishing emails and scam websites to defraud individuals and organizations.
The primary capability of the analyzed Mallas (93.4%) is the generation of sophisticated malware designed to evade detection, which constitutes using AI to develop cyber weapons.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk is caused by human cybercriminals intentionally jailbreaking and deploying LLMs post-deployment to facilitate cybercrime.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
The identification of a widespread underground ecosystem of malicious LLM services (Mallas) highlights how cybercriminals are leveraging AI to automate malware and phishing generation. While this lowers the technical barrier for cyberattacks and poses ongoing economic and security challenges, the threat is currently managed within standard cybersecurity frameworks and does not present an immediate crisis to critical national security functions.
The LLMs function as coding and writing assistants that generate malware and phishing templates based on human prompts, requiring human action to execute the actual cyberattacks.
Limited Risk: The systems are fundamentally general-purpose LLMs and chatbots, which fall under the transparency requirements of Limited Risk, though they are being jailbroken for malicious use.
The primary function of 93.4% of the Mallas is generating malware code, while 41.5% are used for generating phishing emails.
The reports provide detailed, academic-backed evidence from a systematic study by Indiana University Bloomington, analyzing 212 Mallas and over 13,000 listings. The roles of the AI systems, the backend models used, and the specific capabilities (malware, phishing, scam sites) are explicitly documented.
The Mallas could be viewed purely as traditional cybercrime tools that happen to use AI APIs, rather than distinct AI safety failures, though the ease of jailbreaking and availability of uncensored models points to systemic AI safety vulnerabilities.
Technology
Model Jailbreaking Vulnerabilities
Susceptibility of public APIs like GPT-3.5 to jailbreak prompts.
Uncensored Open-Source Models
Availability of models with minimal safety checks, like Pygmalion-13B.
Sophisticated Malware Generation
Mallas generate compilable malware that evades VirusTotal detection.
Data Inputs
NSFW Fine-Tuning Datasets
Use of unfiltered or NSFW datasets to fine-tune open-source models.
Jailbreak Prompt Datasets
Exploitation of specific prompt pairs to bypass API safety filters.
Human Factors
User Susceptibility to Phishing
Vigilant users can still be tricked by highly persuasive phishing emails.
Developer Laissez-Faire Attitude
Hosting platforms providing unrestricted access without guidelines.
Active Exploitation by Criminals
Cybercriminals actively using dark LLMs to automate phishing and fraud.
Process and Methods
Inadequate Hosting Platform Rules
Platforms like FlowGPT lack clear usage guidelines and enforcement.
Bypassing RLHF and Moderation
Mallas easily circumvent standard RLHF and API moderation mechanisms.
Lack of Mandatory Incident Reporting
Organizations lack standardized processes for prompt cyber incident disclosure.
Management
Prioritizing Innovation over Security
Failure to balance rapid AI tool deployment with robust safety features.
Inadequate API Safeguard Auditing
Insufficient testing of public APIs against known jailbreak techniques.
Laissez-Faire Platform Management
Management fails to establish clear usage guidelines on hosting platforms.
Regulatory Environment
Absence of Stricter AI Regulations
Lack of mandated ethical development and robust security standards.
Lack of Global Cyber Cooperation
Absence of unified international frameworks to track and prosecute criminals.
Unrestricted Open-Source Access
Scientific-only models are publicly accessible without safety protocols.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
