This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
In April 2025, an AI-powered customer support bot for the coding assistant Cursor hallucinated a non-existent login policy, falsely informing users they could no longer use the software on multiple devices. This misinformation caused significant user backlash and led to subscription cancellations. The company's leadership later clarified that no such policy existed and attributed the error to the AI bot.
In April 2025, users of Cursor, an AI-powered coding assistant developed by Anysphere, reported being logged out unexpectedly. An AI-powered support bot, "Sam," allegedly responded with an invented login policy to justify the behavior. The hallucinated policy was not based on any real company change. The incident reportedly led to subscription cancellations.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The bot generated and spread false information about company policies, leading to inaccurate beliefs and subscription cancellations.
The AI support bot failed to perform reliably by hallucinating a non-existent company policy and presenting it as fact to customers.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the Cursor support AI bot hallucinating a non-existent policy, which was an unexpected and unintentional outcome after its deployment.
An AI customer support bot for the coding assistant Cursor hallucinated a non-existent login policy, causing user confusion and minor financial loss for the company through subscription cancellations. The incident has negligible national security implications and represents a standard commercial AI reliability issue.
The AI system functioned as a customer support chatbot designed to assist users with inquiries, though it operated without direct human oversight when sending the emails.
Limited Risk: The system is a customer support chatbot. The report notes that the company 'failed to tell users that its customer support person is actually a hallucinating bot', highlighting transparency issues central to Limited Risk systems.
The AI system was deployed as an automated customer support bot to interact with users and answer their queries.
The reports provide clear, consistent details about the specific incident involving Cursor's support bot, including the nature of the hallucination, the company's response, and the user reaction. There is minimal ambiguity regarding the core events.
None
Technology
AI Chatbot Hallucinations
The bot confidently made up a non-existent login policy to explain a bug.
Compounding Errors in Reasoning
Step-by-step reasoning models can compound errors at each thinking step.
Forgetting Tasks via RL
Reinforcement learning focus on some tasks makes the AI forget others.
Data Inputs
Lack of Grounded Truth Data
AI models cannot determine objective truth from their training data.
Training on Uncurated Web Data
Exhausting English internet text led to reliance on unverified source data.
Human Factors
User Deception by Machine
Users felt tricked because the bot posed as a human support agent.
Lack of Human Oversight
No human reviewed the bot's email before it was sent to customers.
Process and Methods
Absence of Guardrails
No constraints were placed on the bot to prevent unsupported claims.
Repetitive AI Canned Responses
Support relied on repetitive AI responses instead of resolving issues.
Management
Failure to Disclose AI Use
Management failed to inform users that support was handled by a bot.
Over-Reliance on Automation
Deploying autonomous bots to save labor costs without risk mitigation.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
