This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A user reported that PayPal's generative AI chatbot falsely informed them of a declined transaction that never occurred. The user was forced to contact human customer support to verify the account status, confirming the chatbot had fabricated the alert. This incident highlights issues with deploying generative AI in financial services without adequate fact-checking or error-handling mechanisms.
On July 13th, 2024, a user reported an incident involving PayPal's generative AI chatbot. The chatbot allegedly incorrectly informed the user of a declined transaction that never occurred, causing confusion and prompting a call to customer service for clarification. This false alert suggests a flaw in the AI system's reliability. The incident created unnecessary labor for both the user and PayPal's human support, demonstrating the potential harm of deploying generative AI without thorough testing and error handling mechanisms.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The chatbot failed to perform reliably by hallucinating data and failing to access actual transaction records.
The chatbot fabricated a declined transaction that never occurred, presenting false information to the user.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the PayPal Assistant chatbot post-deployment, which unintentionally fabricated a declined transaction alert during a user interaction.
Added categories peak at 2: Minor (Epistemic Harm, Inferred)
A PayPal customer service chatbot hallucinated a declined transaction alert, causing minor inconvenience to a user. This incident represents a standard commercial software malfunction with negligible national security implications.
The AI system is described as a 'PayPal Assistant' chatbot designed to help users find information and answer questions, requiring human interaction and oversight.
Limited Risk: The system is a chatbot ('PayPal Assistant'), which falls under Limited Risk due to transparency obligations to ensure users are aware they are interacting with an AI.
The AI system is explicitly described as a generative AI chatbot ('PayPal Assistant') deployed to help users find information on the PayPal site.
The report is a first-hand account of a specific interaction with the PayPal chatbot, clearly detailing the fabricated transaction and the subsequent verification with human customer support. The details are straightforward and unambiguous.
None
Technology
Generative AI Hallucination
The chatbot fabricated a non-existent declined transaction of 23.64 USD.
Inherent LLM Limitations
Generative AI employs randomness and is not designed to look up facts.
Data Inputs
Lack of Verifiable Data Sync
The chatbot did not verify actual database records before claiming a decline.
Human Factors
Deployer Misunderstanding
Deploying AI technology without understanding its core limitations.
Process and Methods
Lack of Product Testing
The AI product was deployed to customers without sufficient testing.
Inactive Feedback Channels
The provided support email was inactive, preventing error reporting.
Management
Premature Deployment Decision
Management deployed the beta chatbot to customers despite known risks.
Poor Customer Support Loop
Support loop redirected users back to the faulty chatbot.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
