This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
An AI startup in Bhutan, NoMindBhutan, created and posted unauthorized promotional deepfake videos of three famous Bhutanese figures, while other residents fell victim to traumatizing deepfake blackmail and intimate image abuse scams.
A TikTok user who posted about missing Bhutan while living abroad reportedly faced blackmail involving manipulated videos described as deepfakes. Scammers allegedly sent the videos to some of his contacts after he initially dismissed their threats. The episode reportedly caused distress. The AI tool and his country of residence were not identified.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
The report mentions non-consensual intimate image-based abuse (NCII), which constitutes exposure to toxic and sexually degrading synthetic content.
The unauthorized replication of the likeness and voices of famous singers and influencers devalues their personal brand and creative identity.
The primary incidents involve using AI deepfakes to impersonate public figures for commercial promotion without consent, and targeting individuals for blackmail and harassment.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The deepfakes were intentionally created and posted by human actors (the startup founders and scammers) using post-deployment open-source AI models.
Added categories peak at 2: Minor (Psychological Harm, Direct)
An AI startup in Bhutan used unauthorized deepfakes of public figures for promotion, while other citizens faced deepfake blackmail and non-consensual intimate image abuse. These incidents highlight growing localized societal risks and privacy violations from accessible AI tools, but carry negligible direct threats to national security, critical infrastructure, or state sovereignty.
The AI models (LLaMA2, Falcon 7b, and deepfake models) were operated locally by humans as tools to perform specific, well-defined functions like face and voice replication.
Limited Risk: The report describes the creation and use of 'AI-generated content such as deepfakes' and 'Chatbots', which are classified as Limited Risk under the EU AI Act, requiring transparency obligations.
The AI systems were primarily used to replicate the faces and voices of individuals to generate deepfake videos.
The report clearly details the specific deepfake incidents, the parties involved, and the resulting psychological and privacy harms. The classification of deepfakes under the EU AI Act and the Causal/Domain taxonomies is straightforward based on the provided text.
The NoMindBhutan incident could be viewed purely as an intellectual property or consent dispute rather than a malicious scam, but the inclusion of blackmail and NCII cases in the report justifies the broader 'Malicious actors' classification.
Technology
Accessible Open-Source AI Models
Easy access to deepfake models via online platforms like Hugging Face.
Advanced Voice and Face Cloning
AI models can replicate faces and voices with high realism and ease.
Data Inputs
Unprotected Social Media Media
Publicly available images and videos are harvested to train deepfakes.
Lack of Consent for Personal Data
Using influencers' faces and voices without obtaining explicit consent.
Human Factors
Disregard for Explicit Consent
Creators assumed social media likes on comments equaled consent.
Lack of Online Skepticism
Users trust malicious links and online content too easily.
Process and Methods
Weak Verification Processes
No formal process to verify consent before publishing AI content.
Insecure Link Clicking
Clicking unknown links led to account freezing and data harvesting.
Management
Prioritizing Growth Over Ethics
Startup prioritized quick deployment over ethical considerations.
Inadequate Risk Management
Management failed to assess the risks of publishing unconsented deepfakes.
Regulatory Environment
Lack of AI Governance laws
Absence of clear regulations on deepfakes and non-consensual media.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
