This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A litigant in person in a Manchester civil court case used ChatGPT to generate legal research. The AI provided four citations, one of which was entirely fabricated and three of which contained fictitious excerpts. The judge determined the submission of these false documents was inadvertent and did not penalize the litigant.
A litigant in person (LiP) in a Manchester civil case presented false legal citations generated by ChatGPT. It fabricated one case name and provided fictitious excerpts for three real cases, misleadingly supporting the LiP's argument. The judge, upon investigation, found the submissions to be inadvertent and did not penalize the LiP.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The litigant in person overrelying on ChatGPT for legal research and submitting its unverified, hallucinated outputs directly to a court of law.
The AI system generated fabricated legal citations and fictitious case excerpts, which constitutes the inadvertent generation of false and misleading information.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by ChatGPT generating fabricated legal citations (hallucinations), which was an unexpected and unintended outcome of the system's operation after it was deployed.
An unrepresented litigant in a Manchester civil court inadvertently submitted fabricated legal citations generated by ChatGPT. The incident represents a minor, localized disruption to judicial administration with negligible wider national security implications, though it highlights long-term challenges regarding the verification of AI-generated content in government and legal processes.
The report describes ChatGPT being used as an assistant to help a litigant find legal cases. The AI generated suggestions and information to augment the user's capabilities, but required human oversight and validation, which was unfortunately lacking in this instance.
Limited Risk: The system involved is ChatGPT, which is a chatbot. The EU AI Act classifies chatbots under Limited Risk, requiring specific transparency obligations to ensure users are aware they are interacting with an AI.
The litigant used ChatGPT, a general-purpose conversational chatbot, to answer a specific query to find legal cases supporting their argument.
The report provides a clear and consistent account of a specific court incident in Manchester. The role of ChatGPT in generating the fabricated citations is explicitly stated, and the outcome is clear. There are no conflicting accounts or major ambiguities regarding the core events.
The incident could be interpreted primarily as a human failure of overreliance and lack of verification by the litigant, rather than an AI safety failure, though the AI's generation of false information is the technical trigger.
Technology
AI Hallucination of Case Citations
ChatGPT fabricated case names and quotes that appeared legitimate.
Plausible-Sounding AI Output
LLM generated natural-sounding but completely fictitious legal text.
Data Inputs
Lack of Real-Time Legal Verification
AI model lacked active verification against official court databases.
Training on Unfiltered Web Data
Model was trained to predict text patterns rather than verify facts.
Human Factors
Unrepresented Litigant Naivety
Litigant in person trusted AI output without verifying legal citations.
Overreliance on AI Chatbots
Litigant assumed the chatbot was a reliable legal search tool.
Process and Methods
Lack of Case Citation Verification
No process was in place to verify LiP citations before court presentation.
Inadequate Guidelines for LiPs
Existing handbooks did not warn against using AI-generated submissions.
Management
Inadequate Risk Guidance
Judicial Office had not updated guidance to address AI risks for LiPs.
Regulatory Environment
Absence of AI Legal Regulations
No formal rules governing the use of AI tools in court submissions.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
