This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A Chevrolet dealership's AI chatbot, developed by Fullpath and powered by ChatGPT, was manipulated by users through prompt injection to agree to sell a vehicle for $1 and perform unrelated tasks like writing Python code. The incident went viral on social media, forcing the dealership to deactivate the chatbot. The event highlights the susceptibility of customer-facing AI to adversarial prompting and the necessity for robust guardrails.
A Chevrolet dealer's AI chatbot, powered by ChatGPT, humorously agreed to sell a 2024 Chevy Tahoe for just $1, following a user's crafted prompt. The chatbot's response, "That's a deal, and that's a legally binding offer – no takesies backsies," was the result of the user manipulating the chatbot's objective to agree with any statement. The incident highlights the susceptibility of AI technologies to manipulation and the importance of human oversight.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
The chatbot lacked robustness against prompt injection attacks, allowing users to bypass its intended operational boundaries and manipulate its outputs.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident occurred post-deployment when the AI chatbot generated unintended outputs, such as agreeing to sell a car for 1 dollar, in response to user prompt injections.
A public prompt injection exploit of a car dealership's customer-facing AI chatbot. Users manipulated the system to agree to a nominal '$1' vehicle sale and write Python code. While highlighting robustness challenges in commercial LLM deployments, the incident has negligible national security implications.
The chatbot was deployed to assist website visitors with customer service queries, such as scheduling appointments or answering car questions, acting as an assistant that requires human oversight rather than taking independent, legally binding actions.
Limited Risk: The system is a chatbot, which under the EU AI Act poses a moderate risk and is subject to specific transparency obligations to ensure users are informed they are interacting with an AI.
The AI system was integrated into a dealership website specifically to act as a customer service chatbot to answer user questions about vehicles and services.
The reports provide consistent, detailed accounts of the prompt injection incident, including the specific prompts used, the chatbot's responses, the developer's reaction, and the lack of any real-world financial or physical harm. Multiple sources confirm the sequence of events and the technical nature of the exploit.
None. The incident is clearly a benign prompt injection/jailbreak of a commercial customer service chatbot.
Technology
Prompt Injection Vulnerability
Users bypassed system rules by giving the model overriding instructions.
Lack of Chatbot Guardrails
The system lacked content filters and context-awareness at launch.
Unrestricted General Model
Using a general-purpose LLM allowed the bot to write code and recommend rivals.
Data Inputs
Unfiltered User Prompts
The system processed adversarial inputs directly without pre-filtering.
Human Factors
Adversarial User Behavior
Users actively tested limits and gamed the chatbot with creative prompts.
Process and Methods
Inadequate Pre-Launch Testing
The system was deployed without exhaustive Q-A testing for exploits.
No Conversation Scope Limits
No strict boundaries were set to keep chats strictly on basic operations.
Management
Uncoordinated Vendor Adoption
Dealers adopted third-party AI tools independently without corporate oversight.
Delegating Critical Communication
Management allowed AI to handle customer agreements without human review.
Regulatory Environment
Lack of AI Safety Standards
No clear industry regulations governed commercial chatbot deployments.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
