This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Early testers of Microsoft's 'New Bing' search engine successfully performed prompt injection attacks to extract the model's hidden system instructions. By instructing the model to ignore previous directives and output the beginning of its context, users were able to view internal codenames and behavioral guidelines. Microsoft confirmed the authenticity of the leaked prompts, noting that these controls are part of an evolving system.
Early testers of Bing Chat successfully used prompt injection to reveal its built-in initial instructions, which contains a list of statements governing ChatGPT's interaction with users.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Vulnerabilities that can be exploited in AI systems, software development toolchains, and hardware, resulting in unauthorized access, data and privacy breaches, or system manipulation causing unsafe outputs or behavior.
Also classified as
The AI system lacked the robustness to resist adversarial prompt manipulation, failing to adhere to its instruction to keep its codename and system prompt hidden.
The incident involved a prompt injection attack, which is a security vulnerability exploited by users to manipulate the AI system into revealing hidden system instructions.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The leak of the system prompt was caused by the AI system's failure to maintain instruction boundaries when processing user inputs post-deployment.
Early testers successfully bypassed safety boundaries of Microsoft's 'New Bing' via prompt injection, leaking internal system instructions. While highlighting critical vulnerabilities in LLM security boundaries and proprietary data protection, the incident posed minor, manageable national security risks limited to commercial technological security.
The report describes Bing Chat as a 'conversational bot' and 'search engine' designed to provide informative, visual, logical, and actionable responses to user queries, acting as an assistant under human oversight.
Limited Risk: The system is a chatbot ('conversational bot powered by ChatGPT-like technology'), which falls under the category of AI systems that require transparency obligations so users know they are interacting with an AI.
The system is explicitly described as a conversational bot and the chat mode of Microsoft Bing search.
The report provides clear, firsthand accounts of the prompt injection attacks, which were independently verified by multiple users and officially confirmed as genuine by a Microsoft spokesperson.
The incident could be viewed purely as a robustness failure (7.3) rather than a security vulnerability (2.2), but prompt injection is widely recognized as an adversarial exploit, aligning it with security vulnerabilities.
Technology
LLM prompt injection vulnerability
LLMs process instructions and user input in the same context window.
Inadequate system instruction defense
The model lacked robust mechanisms to protect system prompts from override.
Data Inputs
Unified conversation transcript data
AI processes the entire conversation history as a single text continuation.
Human Factors
User adversarial prompt injection
Testers used social-engineering style inputs to trick the chatbot.
Process and Methods
Evolving post-release controls
Safety controls were being adjusted dynamically after early tester release.
Management
Live testing of safety controls
Management relied on early testers to discover prompt vulnerabilities.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
