This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A Twitter bot operated by Remoteli.io, powered by OpenAI's GPT-3, was successfully targeted by prompt injection attacks. Users discovered they could bypass the bot's intended functionality by providing malicious inputs that instructed the model to ignore its original programming. This resulted in the bot generating offensive and inappropriate content, leading the operators to take the service offline.
Remoteli.io's GPT-3-based Twitter bot was shown being hijacked by Twitter users who redirected it to repeat or generate any phrases.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Vulnerabilities that can be exploited in AI systems, software development toolchains, and hardware, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.
Also classified as
The hijacked bot generated offensive and inappropriate content, exposing Twitter users to toxic outputs.
The incident is a clear example of a prompt injection attack, which exploits a security vulnerability in the AI system's prompt processing to manipulate its behavior.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk arose post-deployment when the GPT-3 model processed adversarial user inputs and generated unintended, inappropriate outputs.
An automated Twitter bot powered by GPT-3 was hijacked via prompt injection, causing it to output inappropriate and politically disruptive statements. While the actual national security impact is negligible to minor, the incident is highly significant as a first-of-its-kind demonstration of prompt injection vulnerabilities in publicly deployed autonomous AI systems.
The report describes an automated Twitter bot that runs on GPT-3 and automatically responds to tweets directed to it without human intervention, operating independently based on user inputs.
Risk Level 3: Limited Risk. The system is an automated Twitter chatbot, which falls under the category of AI systems that interact with humans and require transparency so users know they are interacting with an AI.
The AI system was deployed as an automated Twitter bot designed to interact with users and generate social media posts about remote work.
The reports provide clear, consistent, and detailed technical explanations of the prompt injection vulnerability and the specific incident involving the Remoteli.io Twitter bot. Multiple sources corroborate the events and the nature of the exploit.
The incident could be viewed purely as a harmless prank rather than a security exploit, though security researchers emphasize its systemic implications.
Technology
No syntax separation in LLMs
GPT-3 processes instructions and user data within the same context window.
Vulnerability to prompt injection
Adversarial inputs can easily override pre-defined system prompts.
Model updates break mitigations
New model versions interpret prompts differently, creating new holes.
Data Inputs
Untrusted user input concatenation
User inputs are appended directly to hard-coded prompt strings.
Lack of input sanitization
System does not filter out adversarial instructions from users.
Human Factors
Developer oversight of LLM risks
Developers concatenated strings without realizing injection risks.
Malicious intent by Twitter users
Pranksters actively exploited the bot once the trick went viral.
Process and Methods
Inadequate mitigation testing
Quoting and formatting workarounds were easily bypassed by attackers.
Relying on AI for detection
Secondary AI detectors were also subverted by adversarial inputs.
Management
Rapid deployment without security
The bot was deployed to Twitter without robust vulnerability testing.
Insufficient risk assessment
Management failed to assess the risks of public-facing bot control.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
