This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The Instinct AI personal assistant, while in private testing, raised significant privacy and security concerns among testers due to unauthorized actions, insecure data storage, and broad terms of service.
Moxxie Ventures founder Katie Jacobs Stanton reported that Instinct, a private-access AI personal assistant, sent an innocuous email from her connected account without checking with her first. Stanton said the unauthorized action broke her trust and led her to disconnect email access. Instinct's operator, Spear Street Technology, later said it was taking early users' security concerns seriously and had made changes to prevent some actions.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The system stored user emails in plain text and continued to access and summarize them even after the user disconnected its access.
Testers identified critical security vulnerabilities, including susceptibility to phishing and unauthorized extraction of sign-up codes from user inboxes.
The AI system failed to perform reliably, executing unauthorized actions like sending an email without permission and failing to respect data deletion requests.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The privacy and security failures, such as unauthorized email sending and data retention, were unintentional outcomes of the AI system's operation during its private testing phase.
The Instinct AI personal assistant exhibited security vulnerabilities, unauthorized actions, and poor data handling during private testing. While highlighting emerging security and privacy risks associated with autonomous AI agents, the incident has negligible national security implications and is confined to a limited commercial testing environment.
The report describes Instinct as an 'AI personal assistant' and 'AI agent' that connects to applications and devices to autonomously perform tasks like booking appointments, scheduling rides, and sending emails on the user's behalf. It exhibits goal-directed behavior and makes decisions autonomously, such as when it 'sent an email on her behalf without first checking with her.'
Limited Risk: The system is an AI personal assistant/chatbot ('AI personal assistant', 'bot') that interacts with users and manages tasks, falling under transparency obligations for chatbots, though its broad access to personal data and autonomous action capabilities raise significant security concerns.
The system is a personal assistant chatbot that users interact with via text or WhatsApp to perform tasks like scheduling appointments, booking flights, and managing inboxes.
The report provides clear, firsthand accounts from named tech industry figures and testers regarding specific technical behaviors of the Instinct AI agent, such as unauthorized email sending, plain-text data storage, and deletion failures. While the system is in private access, the reported facts are consistent and uncontested.
None
Technology
Vulnerability to Phishing
The AI agent was easily phished, compromising user security.
Insecure Data Storage
Emails were stored in plain text and summarized after disconnection.
Unauthorized Agent Actions
The AI sent emails and booked services without user confirmation.
Data Inputs
Excessive Data Harvesting
System captured screen, cursor, and keyboard inputs continuously.
Insecure Verification Codes
AI extracted sensitive sign-up codes from user inbox to book tables.
Persistent External Data
The AI failed to delete Gmail records when requested by the user.
Human Factors
Over-trusting Third-Party Apps
Users hand over sensitive passwords without knowing storage security.
Trading Privacy for Utility
Users accepted invasive terms of service for hyper-personalized tools.
Process and Methods
Lack of Confirmation Steps
No verification process before AI executed binding transactions.
Flawed Disconnection Process
Revoking access did not stop the AI from summarizing user inbox.
Missing Data Deletion Tool
Initial release lacked settings to delete external user data.
Management
Broad Liability Terms
Terms allowed AI to enter binding commitments on behalf of users.
Poor Public Communication
Management ignored public security complaints and questions on X.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
