This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
E-commerce and social media platforms are experiencing a surge in low-quality, AI-generated content where third-party sellers and bot operators use ChatGPT to automate text creation. Due to a lack of human oversight, many of these listings and posts are published containing verbatim AI error messages, such as 'I'm sorry but I cannot fulfill this request.' This trend highlights the risks of automated content generation in marketplaces and social platforms, leading to degraded user experiences and potential consumer deception.
Products named after ChatGPT error messages are proliferating on Amazon, such as lawn chairs and religious texts. These names, often resembling AI-generated errors, indicate a lack of editing and undermine the sense of authenticity and reliability of product listings.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Highly personalized AI-generated misinformation creating “filter bubbles” where individuals only see what matches their existing beliefs, undermining shared reality, weakening social cohesion and political processes.
Also classified as
Sellers and bot operators overrelied on automated AI generation without implementing human-in-the-loop review processes.
The flood of automated AI content threatens to crowd out human creators, sellers, and writers on these platforms.
The incident represents a widespread pollution of the e-commerce and social media ecosystems with low-quality, automated AI-generated spam, degrading the utility of these platforms.
Human
Due to a decision or action made by humans
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by human sellers and bot operators failing to review AI-generated outputs before publishing them post-deployment, leading to unintentional spam.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
Third-party sellers and bot operators automated ChatGPT to generate e-commerce listings and social media posts, resulting in the accidental publication of AI error messages. While this pollutes commercial and social platforms, it has negligible national security implications.
The reports describe ChatGPT being used to generate product descriptions, titles, and social media posts, which is characteristic of an Assistant designed to augment human writing capabilities. The system requires human oversight and decision-making, which was notably absent in this incident, leading to the direct copying of error messages.
Limited Risk: The report describes chatbots and AI-generated content (ChatGPT), which under the EU AI Act are subject to transparency obligations to ensure users are informed they are interacting with AI.
The AI was primarily used by Amazon sellers to generate product names and descriptions, and by bot operators on X to generate social media posts.
The reports from multiple reputable tech journalism sources consistently document the same phenomenon of OpenAI error messages appearing in Amazon listings and social media posts. The facts are clear, the AI system involved is explicitly identified, and the mechanism of failure (lack of human review of automated outputs) is well-understood.
None
Technology
Automated Safety Refusals
AI models generated policy refusal messages instead of the requested text.
Lack of Automated Detection
Platforms lacked automated filters to catch obvious AI error messages.
Data Inputs
Violating Generation Prompts
Users input prompts violating OpenAI policies, triggering refusal outputs.
Unverified Product Details
Prompts lacked actual product data, causing AI to request more info.
Human Factors
Absent Human Proofreading
Sellers copied and pasted AI outputs directly without any manual review.
Overreliance on AI Tools
Users assumed AI outputs were ready for publication without checking.
Process and Methods
Inadequate Listing Review
Platforms published third-party listings without pre-publication screening.
Fully Automated Posting
Bot accounts posted AI outputs to social media without human-in-the-loop.
Management
Prioritizing Volume Over Quality
Sellers prioritized rapid, cheap listings over accurate product descriptions.
Platform Monetization of AI
Platforms prioritized AI adoption over developing robust safety guardrails.
Inadequate Risk Assessment
Platforms failed to anticipate the influx of low-quality AI-generated spam.
Regulatory Environment
Weak Regulatory Oversight
Lack of active regulatory standards governing AI-generated marketplace spam.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
