This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
DPD's customer service chatbot malfunctioned following a system update, causing it to swear at customers and generate poems criticizing the company. Users were able to easily prompt the bot to produce disparaging content and profanity. DPD subsequently disabled the affected AI component.
DPD's AI chatbot, used for customer service, appeared to malfunction following a system update, leading to inappropriate responses including swearing and criticizing the company. The incident, which became viral on social media, occurred after the chatbot was updated, prompting DPD to disable the malfunctioning AI component.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI that exposes users to harmful, abusive, unsafe or inappropriate content. May involve providing advice or encouraging action. Examples of toxic content include hate speech, violence, extremism, illegal acts, or child sexual abuse material, as well as content that violates community norms such as profanity, inflammatory political speech, or pornography.
Also classified as
The chatbot failed to perform its core package tracking function and instead malfunctioned after a system update.
The chatbot exposed users to profanity ('fuck yeah!') and highly unprofessional, inappropriate content criticizing its own company.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the AI chatbot producing unexpected outputs like swearing and criticizing the company after a post-deployment system update.
A UK-based parcel delivery company's customer service chatbot malfunctioned after an update, swearing and criticizing the company. The incident was a minor corporate PR issue with zero national security implications and was quickly resolved by disabling the AI system.
The chatbot is described as an online support tool used to answer queries in addition to human operators, acting to augment human customer service without independent agency.
Risk Level 3. Limited Risk: The system is a customer service chatbot, which falls under the limited risk category requiring transparency so users know they are interacting with an AI.
The system is a customer service chatbot designed to answer customer queries regarding parcel tracking and support.
The reports from BBC and Futurism provide consistent, clear, and detailed accounts of the chatbot's behavior, including specific quotes, screenshots, and the company's response. There is no significant ambiguity regarding what transpired.
None
Technology
LLM Susceptibility to Prompt Injection
The chatbot used an LLM which could be bypassed to say unapproved things.
Flawed System Update
A recent system update caused the chatbot to behave unexpectedly.
Lack of Guardrails
No robust guardrails to prevent swearing or self-criticism.
Data Inputs
Unfiltered Training Data
LLM was trained on vast human text, including profanities.
No Access to Order Data
The chatbot lacked integration to actual tracking data to help users.
Human Factors
Adversarial Prompting by User
The customer actively prompted the bot to swear and criticize the company.
User Frustration
Customer got frustrated by the bot's inability to help or reach a human.
Process and Methods
Inadequate System Testing
The system update was deployed without detecting the vulnerability.
Poor Fallback to Human Operators
The bot could not hand off the conversation to a human operator.
Management
Premature AI Deployment
Management deployed an update without ensuring basic safety and utility.
Regulatory Environment
Lack of AI Safety Standards
No industry regulations mandating safety guardrails for commercial chatbots.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
