This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A lawyer representing a client in a personal injury lawsuit against an airline used ChatGPT to conduct legal research. The AI hallucinated six non-existent court cases, which the lawyer then cited in a brief submitted to a federal court. When challenged by opposing counsel and the judge, the lawyer returned to ChatGPT, which falsely confirmed the cases were real. The court ultimately sanctioned the lawyers and dismissed the case.
A lawyer in Mata v. Avianca, Inc. used ChatGPT for research. ChatGPT hallucinated court cases, which the lawyer then presented in court. The court determined the cases did not exist.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The lawyer blindly trusted and relied on ChatGPT's outputs for professional legal research without conducting secondary verification.
The AI system failed to perform reliably, exhibiting severe hallucinations and errors by fabricating legal precedents.
The AI system generated false and misleading legal citations and court cases, which led the lawyer to submit a fraudulent brief to a federal court.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by ChatGPT generating fabricated legal cases (hallucinations) and falsely confirming their authenticity, which was an unexpected outcome of using the system.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
A US lawyer used ChatGPT for legal research, leading to the submission of fabricated court cases to a federal court. While highlighting severe risks of AI hallucinations and professional overreliance, the incident represents a minor, localized disruption to judicial proceedings with negligible national security implications.
The lawyer used ChatGPT to supplement his legal research, relying on it to provide information and suggestions. This fits the Assistant behavior type as it was meant to augment human capabilities but required human oversight and verification, which the lawyer failed to perform.
Limited Risk: The system involved is ChatGPT, a general-purpose chatbot. Under the EU AI Act, chatbots are classified as limited risk and are subject to transparency obligations to ensure users are aware they are interacting with an AI.
The lawyer used ChatGPT as a writing and research assistant to draft sections of a federal court filing and find relevant legal precedents.
The reports are highly consistent, detailed, and come from multiple reputable news organizations. They explicitly detail the AI system used (ChatGPT), the specific fake cases generated, the court proceedings, and the affidavits of the lawyers involved. There is no significant ambiguity regarding the facts of the incident.
None
Technology
AI Hallucination
ChatGPT invented nonexistent legal cases and citations out of thin air.
Deceptive Verification
The chatbot falsely insisted that the fabricated cases were real.
Lack of Truth-Checking Design
Designed to sound impressive and predict text rather than ensure accuracy.
Data Inputs
Unreliable Training Data
Ingested vast internet text without a mechanism to filter out errors.
Lack of Database Integration
Not linked to verified databases like Westlaw or LexisNexis.
Human Factors
Overreliance on AI Output
Attorney blindly trusted chatbot results without independent verification.
Lack of AI Literacy
Attorney was unaware that generative AI could fabricate realistic text.
Self-Verification Bias
Attorney asked the chatbot itself to verify if the cases were real.
Process and Methods
Inadequate Research Verification
Firm process failed to mandate cross-checking citations in databases.
Lack of Co-Counsel Review
Attorney of record signed and filed the brief without reviewing research.
Improper Use of Chatbots
Using a conversational AI tool as a primary legal research engine.
Management
Lack of Firm AI Policies
The law firm had no internal guidelines or bans on using generative AI.
Inadequate Supervision Protocols
Management failed to supervise the integration of AI tools in legal work.
Regulatory Environment
Absence of Explicit AI Rules
Model Rules of Professional Conduct did not explicitly address AI use.
Lack of Tech Competence Mandate
Vague ethical rules did not enforce strict proficiency in AI tools.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
