This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Legal professionals in both Canada and the United States submitted court filings containing fabricated legal citations generated by AI tools. These incidents resulted in judicial sanctions, including fines and contempt of court proceedings, as the lawyers failed to verify the accuracy of the AI-generated research before submitting it to the courts.
In Ko v. Li, 2025 ONSC 2766, the Ontario Superior Court of Justice identified multiple inaccurate or nonexistent case citations in a factum submitted by counsel for the applicant. Justice FL Myers questioned whether generative AI was used in drafting the document, noting the cited cases were either irrelevant, misrepresented, or did not exist. The court ordered the lawyer to show cause why she should not be cited for contempt, citing duties of accuracy and technological competence.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The lawyers over-relied on the AI tools to conduct legal research and generate briefs without verifying the accuracy of the citations.
The AI systems generated false and misleading legal citations (hallucinations) that were presented as factual precedents in court filings.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by AI systems generating fabricated legal citations, which was an unintentional and unexpected outcome of their deployment.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
Lawyers in the US and Canada submitted court filings containing fabricated legal citations generated by AI tools, resulting in judicial sanctions. This incident represents a minor national security concern, specifically affecting sovereignty and government functions by temporarily disrupting court proceedings, but was easily managed through standard judicial oversight and disciplinary mechanisms.
The AI systems (CoCounsel, Westlaw Precision, Google Gemini, ChatGPT) were used as assistants to outline, generate, and research legal briefs, which required human verification and oversight that the lawyers failed to provide.
Risk Level 3: Limited Risk. The AI systems used, such as Google Gemini and ChatGPT, are chatbots and content generation tools that pose a moderate risk and require transparency, particularly regarding their propensity to generate hallucinated content.
The AI systems were used by lawyers as writing and research assistants to generate outlines, draft legal briefs, and find case citations.
The reports provide clear, detailed accounts of the two legal cases, including specific court citations, names of law firms, the specific AI tools used, and the exact nature of the hallucinations and subsequent judicial sanctions.
None
Technology
Generative AI Hallucinations
AI tools fabricated non-existent legal citations and hyperlinked them.
Inaccurate Search Results
AI tools mapped real case names to unrelated or opposite legal propositions.
Lack of AI Origin Tracking
AI tools did not clearly flag or watermark generated text as synthetic.
Data Inputs
Unverified Hyperlinks
AI generated plausible-looking hyperlinks that led to 404 errors or wrong pages.
Incomplete Legal Databases
AI models trained on incomplete datasets hallucinated missing legal precedents.
Flawed Reference Mapping
AI mapped citations to incorrect external URLs or unrelated database entries.
Human Factors
Overreliance on AI Outputs
Lawyers assumed AI-generated research was accurate without manual verification.
Failure to Verify Citations
Attorneys submitted briefs without reading the cited cases to ensure existence.
Failure to Disclose AI Use
Attorney shared AI outlines with colleagues without disclosing the AI source.
Process and Methods
Lack of Human-in-the-Loop Review
No process was in place to verify the validity of AI-generated legal research.
Inadequate Quality Control
Corrected briefs were submitted with remaining AI errors due to poor review.
Absence of AI Usage Guidelines
Law firms lacked standardized procedures for integrating AI into legal drafting.
Management
Inadequate Staff Supervision
Partners failed to supervise associates or clerks using generative AI tools.
Failure to Educate Staff on AI Risks
Firms did not train lawyers on the risks of generative AI hallucinations.
Regulatory Environment
Ineffective Professional Oversight
Bar association rules failed to deter lawyers from submitting fake cases.
Lack of AI Auditing Standards
No formal regulatory standards exist for validating AI legal research tools.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
