This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The AI personal assistant Instinct faced privacy and security concerns during private testing after users discovered it retained deleted data, stored emails in plain text, and performed unauthorized actions like sending emails without consent.
AI product founder Claire Vo reported that three hours after she disconnected Instinct from her Google account, the personal AI assistant summarized previously indexed emails. Vo said the Google connector was off, but Instinct had retained copies for later searching; reporting indicates the last stored email predated disconnection. The episode involved the agent acting on retained email data after Google access was revoked, not confirmed continued access to Vo's Gmail account.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
Also classified as
Testers identified that the system was highly vulnerable to phishing and could easily pull sensitive sign-up codes from user inboxes.
The AI system exhibited a lack of robustness by performing unauthorized actions, such as sending an email on a user's behalf without prior confirmation.
The primary issue involves the AI system retaining copies of private emails in plain text and failing to delete user records even after access was revoked, directly compromising user privacy.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The privacy and security risks arose from the AI system's actions, such as retaining data after disconnection and sending unauthorized emails, which were unintentional outcomes of its operation.
During private beta testing of the Instinct AI personal assistant, users discovered privacy and security flaws, including data retention after deletion requests and unauthorized email sending. This incident represents standard commercial software security and privacy challenges with negligible implications for national security.
The system is described as an 'AI personal assistant' and 'AI agent' that can autonomously perform tasks such as booking appointments, making purchases, and sending emails on the user's behalf without requiring step-by-step human intervention.
Limited Risk: The system is described as an 'AI personal assistant' and a 'chatbot' that interacts with users to perform tasks, which falls under the category of chatbots requiring transparency under the EU AI Act.
The system is an AI personal assistant that users interact with via text or WhatsApp to perform tasks like scheduling, booking, and email management.
The reports from TechCrunch and Norton provide clear, consistent details about the AI system's functionality, its terms of service, and specific user experiences highlighting privacy and security issues. There is little ambiguity about what the system did (retaining data, sending unauthorized emails) and how users reacted.
None. The reports consistently frame the incident as a privacy and security risk associated with a highly autonomous AI agent.
Technology
Vulnerability to Phishing
The agent was easily phished, leading to user account deletion.
Insecure Plain Text Storage
Emails were stored in plain text for searches even after disconnection.
No Action Confirmation System
The system sent emails on behalf of users without checking first.
Data Inputs
Intrusive Device Data Collection
Collected screen captures, cursor movements, and keyboard inputs.
Persistent Data Retention
Retained copies of emails and external inputs after disconnection.
Default Model Training Use
Personal data was used for model training by default without consent.
Human Factors
Unaware of Storage Risks
Users handed over passwords without knowing how they would be stored.
Lack of Opt-Out Awareness
Users did not realize they had to manually opt out of data training.
Process and Methods
Flawed Deletion Process
Lacked a tool to delete external Gmail records when requested by users.
No Verification Workflows
Allowed binding commitments and purchases without explicit user confirmation.
Silent Background Processing
Continued summarizing inboxes after email access was disconnected.
Management
Growth Over Security Focus
Prioritized rapid deployment of autonomous features over safety safeguards.
Unfavorable Liability Terms
Shifted all transaction dispute liabilities entirely onto the users.
Delayed Public Response
Management kept a low profile and did not address early public concerns.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
