This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Scammers are leveraging AI tools like ChatGPT to create high-quality, error-free marketing copy and translate stolen content for fraudulent e-commerce listings. These copycats impersonate small businesses by scraping their videos, logos, and social media posts to sell cheap knockoffs or steal money directly from consumers. The resulting fraud has caused significant financial losses and reputational harm to small businesses, who struggle to keep pace with the scale and sophistication of these AI-enabled attacks.
Scammers are using AI to impersonate small businesses by copying their videos, logos, and social media posts. They create fake listings and ads, diverting customers to cheap knockoffs or stealing their money. This has severely impacted businesses like Bee Cups, Darn Tough Vermont, and Cascade hummingbird feeders, leading to significant financial losses, negative reviews, and damaged reputations. Their deployment of AI makes it challenging for small businesses to combat these fraudulent activities.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
Scammers copied and reproduced the creative products and marketing materials of human creators, devaluing their effort and causing massive financial damage.
The incident involves scammers using AI to generate high-quality, error-free copy and translate stolen content to execute highly convincing e-commerce fraud and impersonation scams targeting specific small businesses and their customers.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk is caused by human scammers intentionally using deployed AI systems like ChatGPT to generate error-free copy and translate stolen content for fraudulent e-commerce listings.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
This incident involves scammers using generative AI to scale up e-commerce fraud and impersonate small businesses. While causing notable financial harm to individual companies and consumers, the national security impact remains minor and is manageable within standard law enforcement and regulatory frameworks.
The report describes ChatGPT being used as a tool by scammers to perform specific, well-defined functions such as writing error-free copy and translating text. The AI operates on direct input without autonomous planning or decision-making.
Limited Risk: The AI system (ChatGPT) is a general-purpose chatbot used to generate text content. Under the EU AI Act, such systems are subject to transparency obligations to ensure users are aware they are interacting with AI or that the content is AI-generated.
The scammers used ChatGPT as a writing assistant to generate error-free marketing copy and translate stolen content into other languages like Dutch and Swedish.
The report provides clear, detailed accounts of how scammers use AI tools like ChatGPT to facilitate e-commerce fraud. The financial impacts, specific businesses affected, and the mechanisms of the scams are well-documented. The role of AI is explicitly stated as a tool to avoid language errors and translate content.
The incident could be viewed purely as trademark infringement and counterfeiting, but the explicit mention of AI tools lowering the barrier to entry for high-quality fraud justifies its classification as an AI-enabled safety incident.
Technology
Generative AI Text Generation
Scammers use ChatGPT to write error-free text, hiding typical fraud signals.
Automated Website Replicators
Technology allows easy scraping and replication of legitimate brand websites.
Targeted Ad Algorithms
Scammers use platform algorithms to target customers of legitimate brands.
Data Inputs
Scraped Brand Assets
Scammers copy original photos, logos, and videos to build fake sites.
Manipulated Search Keywords
Scammers bid on brand keywords to divert search traffic to fake listings.
Human Factors
Consumer Inability to Detect Fakes
Consumers struggle to distinguish between real and highly-polished fake ads.
Limited Small Business Resources
Small business owners lack time and money to monitor and fight copycats.
Process and Methods
Slow Platform Takedown Processes
E-commerce and social platforms take days or weeks to remove reported fakes.
Inadequate Ad Screening
Platforms fail to proactively detect and block deceptive ads before publishing.
Management
Inadequate Proactive Monitoring
Small business owners did not anticipate being targeted at such scale.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
