This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Microsoft's Bing Chat feature was observed generating and propagating misinformation by citing previously documented AI-generated disinformation as factual evidence. The system failed to distinguish between reliable news sources and AI-generated content, creating a self-reinforcing loop of false information. This incident highlights the risks of deploying large language models in search contexts without robust mechanisms to verify the accuracy of cited information.
Reporters from TechCrunch issued a query to Microsoft Bing's ChatGPT feature, which cited an earlier example of ChatGPT disinformation discussed in a news article to substantiate the disinformation.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The reports highlight the systemic risk of chatbots consuming and regurgitating each other's AI-generated errors, degrading the broader information ecosystem.
The underlying failure is the AI's inability to reliably distinguish between real news, jokes, and AI-generated disinformation.
The incident primarily involves Bing Chat generating and spreading false or misleading information regarding vaccine safety and competitor product status.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The misinformation was generated by the Bing Chat AI system post-deployment as an unintended consequence of its search and summarization goals.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
Microsoft's Bing Chat propagated misinformation by citing ChatGPT-generated disinformation as factual. While the incident itself had negligible direct national security impact, it highlights an evolved strategic vulnerability where AI systems can create self-reinforcing loops of false information, potentially degrading the broader information ecosystem.
The report describes Bing Chat as a conversational search engine designed to assist users in finding information. It operates on direct user prompts and requires human oversight, as it 'generated — well, plagiarized — the entire thing pretty much in good faith' without independent agency.
Risk Level 3. Limited Risk: The system is a chatbot, which falls under the category of 'AI systems that pose a moderate risk and require specific transparency obligations to ensure users are informed and aware of the AI's nature' such as 'Chatbots'.
The primary function of Bing Chat is to act as an interactive chatbot that performs content search and conversational question answering for users.
The reports provide clear, first-hand accounts of specific interactions with Bing Chat, detailing the exact prompts and outputs. The mechanics of the errors (citing ChatGPT-generated text and Hacker News jokes) are well-documented and verified by tech journalists.
None.
Technology
Self-Reinforcing Feedback Loops
AI models absorb and regurgitate other AI-generated content recursively.
Inability to Verify Factuality
Chatbots cannot reliably distinguish between real facts and fake news.
Vulnerability to Jailbreaking
Simple prompt engineering bypasses safety filters using creative logic.
Data Inputs
Ingestion of AI-Generated Content
Web scrapers ingest synthetic data and misinformation from other AI models.
Lack of Contextual Understanding
Models fail to parse context labeling text as debunked or fake.
Unreliable Source Ingestion
Systems treat joke comments and tweets as authoritative news sources.
Human Factors
Malicious Prompt Engineering
Users intentionally manipulate models to bypass safety protocols.
User Reliance on Chatbots for News
Users treat conversational agents as reliable search engines.
Misinterpretation of AI Output
Users take hallucinated facts or satire as absolute truth.
Process and Methods
Rushed Product Launches
Releasing conversational search tools without adequate safety testing.
Inadequate Filtering of Output
Lack of warnings on controversial, medical, or unverified claims.
Ineffective Source Evaluation
No robust process to evaluate the credibility of cited web sources.
Management
Prioritizing Market Share Over Safety
Big Tech companies rushed chatbot deployment to compete for market share.
Inadequate Risk Assessment
Deploying systems despite known risks of misinformation generation.
Regulatory Environment
Lack of AI Safety Standards
No clear regulatory frameworks governing AI-generated misinformation.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
