This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
New York City launched a Microsoft-powered AI chatbot as part of its 'MyCity' project to assist business owners with regulations and compliance. Investigations by The Markup and other outlets revealed that the bot frequently provided inaccurate and illegal advice, including incorrect information regarding housing discrimination, worker tip laws, and cash-payment mandates. The city has acknowledged the issues, characterizing the tool as a pilot program that requires ongoing improvement.
New York City's chatbot, launched under Mayor Eric Adams's plan to assist businesses, has been reportedly providing dangerously inaccurate legal advice. The Microsoft-powered bot allegedly informed users that landlords can refuse Section 8 vouchers and that businesses can operate cash-free, among other falsehoods. The city acknowledges the chatbot is a pilot program and commits to improvements while the errors are addressed.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The chatbot's false advice encouraged illegal discrimination against housing voucher holders and unbanked individuals.
The chatbot exhibited a lack of capability and robustness by hallucinating false legal information and failing to perform reliably.
The AI chatbot generated and spread incorrect and illegal legal advice to users, leading to inaccurate beliefs about city regulations.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incorrect and illegal advice was generated by the AI chatbot after its deployment, representing an unintended failure of the system's capability.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
A municipal AI chatbot deployed by New York City provided inaccurate and illegal advice regarding local business, labor, and housing regulations. While demonstrating the risks of deploying unverified LLMs in government services, the incident has negligible national security implications and presents no threat to physical security, intelligence, or sovereign stability.
The chatbot acts as an assistant by providing information and suggestions to users to help them start and operate businesses, requiring human oversight and validation.
Limited Risk: The system is an AI chatbot, which falls under Limited Risk due to transparency obligations to ensure users know they are interacting with an AI.
The system was explicitly deployed as an AI-powered chatbot to answer user questions about starting and operating a business in NYC.
The reports provide clear, consistent, and detailed examples of the chatbot's specific incorrect outputs, verified by multiple journalists and experts. The role of the AI system and its developer are explicitly stated.
None
Technology
LLM Hallucinations
Chatbot produces inconsistent and inaccurate responses to users.
Inconsistent Bot Outputs
The bot returns different answers to the same query over time.
Data Inputs
Incomplete Training Data
Bot lacked synthesized knowledge of specific local laws and exceptions.
Lack of Source Citations
Bot often failed to provide verification links for its answers.
Human Factors
Overreliance on Bot Authority
Users trust the bot because it is hosted on an official city website.
Process and Methods
Inadequate Pre-launch Testing
The chatbot was deployed publicly with major legal accuracy gaps.
Deficient Safeguard Mechanisms
No real-time filters to block illegal or harmful business advice.
Management
Premature Pilot Deployment
City launched a live pilot, relying on public testing to find errors.
Prioritizing Launch over Safety
Focus on MyCity portal centerpiece over ensuring legal compliance.
Regulatory Environment
Absence of Municipal AI Standards
No strict regulations governing the deployment of public-facing city bots.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
