This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A businessman in China was defrauded of 4.3 million yuan after a scammer used AI-generated deepfake technology to impersonate a trusted friend during a video call. In a separate incident mentioned in the same report, a man was investigated by police for using ChatGPT to generate and disseminate a fake news article regarding a fatal bus crash. These events highlight the misuse of AI for financial fraud and the spread of disinformation.
A scammer in China used AI-generated deepfake technology to impersonate a businessman’s trusted friend in a video call, convincing him to transfer 4.3 million yuan ($612,000). The fraudster mimicked the friend’s face and voice, claiming another associate needed funds for a business transaction. Without verifying the transfer, the victim sent the money. He later realized the deception and contacted police, who managed to recover 3.4 million yuan.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
The report also describes a man using ChatGPT to generate and spread a fake news article about a deadly bus crash.
The primary incident involves a scammer using AI deepfake technology to impersonate a trusted friend and defraud a businessman of 4.3 million yuan.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incidents were caused by human actors intentionally misusing deployed AI technologies (deepfakes and ChatGPT) to commit financial fraud and spread disinformation.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
The incidents involve the criminal misuse of generative AI in China, specifically real-time deepfake technology to execute a 4.3 million yuan financial fraud and ChatGPT to generate fake news about a bus crash. While demonstrating the evolving sophistication of AI-enabled crime and localized disinformation, the events represent minor domestic law enforcement challenges rather than significant threats to national security.
The AI systems (deepfake face/voice changer and ChatGPT) operated as tools under the direct control of human users to perform specific tasks (generating a fake news article and altering video/audio in real-time) without exhibiting autonomous decision-making or planning.
Limited Risk: The report describes the use of chatbots (ChatGPT) and AI-generated content such as deepfakes, which are classified as limited risk under the EU AI Act and subject to transparency obligations.
The primary AI system was used to generate a real-time deepfake video and voice clone to impersonate a friend, while the secondary system (ChatGPT) was used as a writing assistant to generate a fake news article.
The report provides clear, factual details about the deepfake scam, including the specific amount of money stolen and recovered, the location, and the method of deception. It also clearly details the separate ChatGPT incident. There is little ambiguity regarding the role of AI in these events.
None
Technology
AI Face and Voice Cloning
Con artists used smart AI technology to change their face and voice.
Generative AI Accessibility
Abuse of generative tools for creating highly convincing fake media.
Data Inputs
Fraudulent Payment Records
Scammer sent a fake screenshot of a payment record to deceive the victim.
Stolen Identity Data
Scammer used the friend's personal data to impersonate him convincingly.
Human Factors
Overreliance on Visuals
Victim let down guard after verifying the face and voice on video call.
Failure to Out-of-Band Verify
Victim did not check bank account or call friend before sending money.
Process and Methods
Inadequate Transaction Verification
No robust process to verify large transfers before executing them.
Lack of Deepfake Detection Tools
Absence of real-time deepfake detection during the video call.
Management
Inadequate Corporate Risk Controls
Company account lacked strict multi-authorization protocols for transfers.
Regulatory Environment
Lagging Deepfake Regulations
Regulations on deepfakes and AI safety assessments are still emerging.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
