This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Anthropic customers report that the company's AI-powered customer support chatbot, Fin, frequently fails to resolve billing and subscription issues, trapping users in circular conversations. Multiple users described being unable to reach human support, resulting in prolonged financial overcharges and, in one instance, the accidental cancellation of a business account. The reports highlight concerns regarding the lack of transparency and human oversight in AI-driven customer service systems.
An executive at a Canadian engineering firm alleged that Anthropic's Fin AI support chatbot kept him in circular conversations for weeks while he sought a refund for a $3,500 duplicate Claude AI subscription charge. He later sought a chargeback; Anthropic reportedly canceled the working team account, disrupting work and requiring another subscription purchase. The report does not establish that AI caused the charge or cancellation.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
Anthropic delegated customer support and billing dispute decisions to the AI system, leaving customers with no meaningful human recourse or appeal when errors occurred.
The AI customer support chatbot failed to perform reliably, trapping users in circular loops and failing to escalate billing errors as designed.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The financial and operational harms were caused by the AI chatbot failing to perform its customer service and escalation functions correctly after being deployed.
Added categories peak at 2: Minor (Psychological Harm, Indirect)
Anthropic's customer support chatbot, Fin, failed to resolve billing issues, trapping users in loops and causing minor financial losses and operational disruptions for commercial customers across multiple nations. The incident represents a minor economic and technological security concern related to AI deployment robustness, but poses no threat to national security, critical infrastructure, or sovereignty.
The AI system operates as a customer service chatbot, interacting with users to answer questions and resolve billing issues, which fits the Assistant profile as it is designed to handle customer queries but lacks true independent planning or goal-directed behavior.
Limited Risk: The report describes an AI customer service chatbot. Under the EU AI Act, chatbots are classified as limited risk and are subject to specific transparency obligations to ensure users know they are interacting with an AI.
The primary function of Salesforce Fin is to act as an automated customer service chatbot to resolve user queries. It also uses sentiment analysis to detect customer frustration and determine when to escalate cases.
The report provides clear, firsthand accounts from multiple named and anonymous sources detailing their specific financial losses and interactions with the AI chatbot. The role of the AI system (Salesforce Fin) and the deployer (Anthropic) is explicitly stated, and the failure modes are well-documented with screenshots and direct quotes.
The billing errors themselves could be viewed as standard database or software glitches rather than AI failures, but the core of the reported incident is the AI chatbot's failure to resolve or escalate these errors.
Technology
Non-deterministic AI failures
AI agents unpredictably failed to route customer chats to human support.
Flawed sentiment detection
The chatbot failed to accurately flag frustrated users for escalation.
Data Inputs
Duplicate account creation
System created duplicate profiles, feeding incorrect billing data to AI.
Inaccurate billing inputs
Incorrect subscription plans were fed into the AI customer support system.
Human Factors
Overreliance on AI agents
Anthropic relied almost entirely on AI chatbots to handle billing issues.
Process and Methods
Flawed escalation rules
Rules failed to properly route complex billing issues to human agents.
Automated account lockout
Chargeback requests triggered automated systems that shut down active accounts.
Management
Cost-saving over safety
Anthropic prioritized cheap AI resolutions over maintaining human support.
Inadequate risk assessment
Management failed to assess risks of AI failures on customer operations.
Regulatory Environment
Absence of chatbot laws
No active laws mandated human escalation paths for AI customer service.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
