This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Research by the consumer organisation Which? revealed that several major AI chatbots, including ChatGPT, Copilot, Gemini, Meta AI, and Perplexity, provided inaccurate or misleading financial, tax, and insurance advice to UK users. The systems incorrectly advised on ISA investment limits, tax rules, and travel insurance requirements, and directed users toward costly private services instead of free government alternatives. These inaccuracies pose risks of financial loss and legal complications for consumers relying on the tools for professional guidance.
Several major AI chatbots, including ChatGPT, Copilot, Gemini, and Meta AI, were reportedly found to have provided incorrect or misleading financial and insurance guidance for UK users. The systems allegedly advised exceeding ISA limits, misstated tax rules, gave wrong travel insurance requirements, and pointed users toward costly refund services.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The AI systems failed to perform reliably and accurately when queried on domain-specific financial and legal regulations, demonstrating a lack of capability in critical applications.
The chatbots generated incorrect and misleading financial, tax, and legal advice, which can lead to inaccurate beliefs and potential material or financial harm for users who rely on them.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk is caused by the AI systems generating inaccurate and misleading financial advice, which is an unexpected and unintentional failure mode of the deployed models.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
Major AI chatbots were found to provide inaccurate and misleading financial, tax, and legal advice to UK consumers. While this poses consumer protection challenges and potential minor personal financial risks, it does not represent a significant threat to national security, critical infrastructure, or state sovereignty.
The chatbots act as conversational assistants designed to provide information and suggestions to users, but they require human oversight and verification as they do not take independent actions.
Limited Risk: The systems described are general-purpose chatbots, which are subject to specific transparency obligations under the EU AI Act to ensure users are aware they are interacting with an AI.
The AI systems are general-purpose conversational chatbots used by consumers to ask questions and receive financial, tax, and legal advice.
The report provides clear, factual details about the testing conducted by Which? and the specific incorrect outputs generated by each chatbot. It also includes first-hand user testimony from a business owner who experienced the issue, making the assessment highly reliable.
None
Technology
Hallucination and Logic Errors
Chatbots generated inaccurate tax, insurance, and contract advice.
Failure to Detect User Errors
Models failed to identify and correct the false 25k ISA allowance premise.
Data Inputs
Out-of-date Training Data
ChatGPT used an outdated tax code, leading to incorrect calculations.
Unfiltered Promotional Links
Search results prioritized high-fee premium tax-refund firms over HMRC.
Human Factors
Overreliance on AI Advice
Consumers blindly trusted chatbot advice for critical financial decisions.
Lack of User Verification
Users failed to double-check chatbot outputs with official sources.
Process and Methods
Inadequate Output Verification
Lack of automated accuracy checks for high-risk financial queries.
Lack of Domain-Specific Guardrails
General-purpose models lack specialized constraints for financial advice.
Management
Prioritizing Capability over Safety
Deploying general tools for high-stakes tasks without robust safeguards.
Shifting Verification to Users
Disclaimers used to deflect liability instead of fixing accuracy issues.
Regulatory Environment
Lack of Regulatory Coverage
FCA noted AI advice is not covered by the Financial Ombudsman Service.
Absence of Chatbot Standards
No strict legal standards governing AI-generated financial recommendations.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
