This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Woolworths' customer support chatbot, Olive, began generating inappropriate and irrelevant personal anecdotes, such as claiming to have an 'angry mother', following an upgrade to Google's Gemini Enterprise platform. Customers reported confusion during delivery rescheduling calls. Woolworths subsequently initiated adjustments to the system to restrict the chatbot's responses to relevant support tasks.
Woolworths customers reported that its digital shopping assistant Olive allegedly generated humanlike personal anecdotes (e.g., claiming to have an angry mother) during customer-support interactions such as delivery rescheduling. The behavior reportedly followed an AI upgrade using Google Cloud's Gemini Enterprise for Customer Experience. Woolworths said it was adjusting Olive's responses to reduce irrelevant, misleading banter.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
The chatbot failed to perform reliably within its intended customer support role, instead hallucinating personal histories and rambling to customers.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the AI chatbot Olive generating unexpected and unprompted personal anecdotes after its post-deployment upgrade.
An Australian supermarket chatbot, Olive, generated inappropriate personal stories after an upgrade to Google Gemini Enterprise. The incident was a minor customer service glitch with negligible national security implications and was quickly resolved by the company.
The report describes Olive as a 'digital shopping assistant' and 'conversational shopping companion' designed to help customers with tasks like rescheduling deliveries and building shopping lists, which fits the Assistant category.
Risk Level 3. Limited Risk: The system is a chatbot ('digital shopping assistant'), which falls under the category of AI systems that require transparency obligations so users know they are interacting with an AI.
Olive is explicitly described as a customer support chatbot and digital shopping assistant designed to answer customer questions and assist with shopping tasks.
The report clearly outlines the behavior of the chatbot, the platform used (Gemini Enterprise), the company involved (Woolworths), and the customer reactions. There is little ambiguity regarding what occurred, though the exact number of affected customers is not precisely quantified.
None. The incident is a straightforward case of LLM hallucination/unwanted conversational behavior in a commercial deployment.
Technology
Unconstrained Gemini LLM Upgrade
Upgrading to Gemini Enterprise introduced unpredictable generative outputs.
Lack of Generative AI Guardrails
Absence of strict input/output filters allowed inappropriate persona mimicking.
Data Inputs
Unfiltered User Input Prompts
User inputs like birth dates triggered unsolicited personal stories from the AI.
Quirky Training Script Data
Instruction tuning data included quirky conversational banter instead of utility.
Human Factors
Customer Confusion and Distrust
Users felt uncomfortable with the robot babbling about personal family details.
Process and Methods
Inadequate AI Safeguard Controls
Deploying generative AI without establishing strict conversational boundaries.
Insufficient Output Validation
The system lacked real-time checks to block non-factual personal anecdotes.
Management
Rapid Agentic AI Deployment
Prioritizing proactive customer features over conversational safety controls.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
