This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The Facemega app, developed by Ufoto Limited, was used to run hundreds of sexually suggestive advertisements on Meta platforms and other apps. These ads featured deepfake face-swaps of celebrities like Emma Watson and Scarlett Johansson to promote the app's ability to generate nonconsensual sexual content. Following public outcry and media reports, the app was removed from the Apple App Store and Meta's advertising services, though it remained available on Google Play for a period.
Sexually suggestive videos featuring faces of female celebrities such as Emma Watson and Scarlett Johansson were rolled out as ads on social media for an app allowing users to create deepfakes.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The app was marketed and used to generate humiliating, nonconsensual sexual imagery of specific individuals for personal or commercial advantage.
The incident involved running hundreds of public advertisements featuring sexually suggestive deepfakes of celebrities, exposing millions of social media users to toxic and inappropriate content.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by human developers intentionally creating and running suggestive deepfake ad campaigns to promote their deployed AI application.
Added categories peak at 4: Severe (Epistemic Harm, Inferred)
A commercial deepfake application, Facemega, ran aggressive ad campaigns featuring nonconsensual celebrity face-swaps on major social platforms. While highlighting the rapid commercialization and accessibility of realistic synthetic media tools, the incident represents a minor national security threat, primarily impacting digital privacy, platform safety, and consumer protection rather than critical state infrastructure or functions.
The Facemega app operates as a tool that performs a specific, deterministic function (face-swapping) based on direct user inputs (uploading a photo and selecting a template) without autonomous planning or context retention.
Limited Risk: The report describes an AI system used to generate deepfakes, which falls under the Limited Risk category of the EU AI Act, requiring specific transparency obligations to inform users.
The primary function of the Facemega app is to allow users to swap faces into video templates, generating deepfake videos.
The incident is documented consistently across multiple reputable investigative reports. The details regarding the ad campaign, the platforms involved, the specific celebrities targeted, and the subsequent platform removals are clear and unambiguous.
None.
Technology
Rapid AI Face-Swap Accessibility
Users can generate realistic deepfakes in seconds using just one photo.
Realistic Synthetic Media
Advanced algorithms make fake videos look convincingly real, blurring reality.
Low Barrier to Tech Access
The technology is easily available via free or cheap smartphone apps.
Data Inputs
Unrestricted Photo Inputs
The app allows uploading any single photo of anyone without consent.
Pre-existing Video Templates
App provides ready-made templates, including suggestive or 'hot' videos.
Exploitation of Public Images
Celebrity promotional photos are easily harvested to seed deepfakes.
Human Factors
Malicious User Intent
Users deliberately use the tech to harass, humiliate, and violate women.
Exploding Consumer Demand
High demand for nonconsensual porn fuels the spread of these apps.
Disregard for Consent
Users readily create sexual content of others without their permission.
Process and Methods
Flawed Ad Moderation Systems
Meta, Apple, and Google algorithms failed to block sexually suggestive ads.
Inadequate App Store Vetting
App stores hosted the deepfake app despite policies against harmful content.
Ineffective App-Level Controls
The app failed to enforce its own terms prohibiting explicit content.
Management
Reactive Safety Enforcement
Platforms only removed ads and apps after media exposure by journalists.
Profitization of Harmful Apps
Developers prioritized weekly subscription revenue over safety safeguards.
Lack of Proactive Risk Audits
Management failed to assess risks of deepfake tech before deployment.
Regulatory Environment
Fragmented Legal Frameworks
Only a few jurisdictions explicitly outlaw nonconsensual deepfakes.
Jurisdictional Hurdles
Taking legal action against Chinese parent companies is practically difficult.
Slow Pace of Legislation
Technology advances much faster than the creation of protective laws.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
