This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Two law firms were sanctioned $31,100 by a Special Master after submitting legal briefs containing AI-generated hallucinations, including non-existent case citations and fabricated quotations. The attorneys failed to verify the AI-generated content and subsequently resubmitted a revised brief that still contained errors, without disclosing the use of AI. The court struck the briefs and imposed financial sanctions to deter similar conduct in the legal profession.
In the case of Lacey v. State Farm, two law firms were sanctioned $31,000 after submitting a legal brief containing reportedly erroneous citations generated using AI tools. The court reportedly found that the lawyers failed to disclose the use of AI, neglected to verify its output, and refiled a revised brief with additional inaccuracies. Judge Michael Wilner deemed the conduct reckless and issued sanctions for what he described as "improper" and "misleading" legal filings.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The attorneys overrelied on the AI tool by failing to verify or cite-check the generated research before submitting it to the court.
The AI system generated false and misleading legal citations and quotations, which were then submitted to a court, leading to inaccurate beliefs and disrupting the judicial process.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by an AI system generating hallucinated legal citations, which was an unintentional and unexpected outcome of using the tool for research post-deployment.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
Attorneys in a federal civil case were sanctioned for submitting briefs containing AI-generated fake citations. While this compromised the integrity of a local judicial proceeding, the national security impact is negligible, representing a minor, manageable disruption to government functions with no broader threat to physical, economic, or information security.
The report describes the attorney using AI tools to generate an 'outline' for a supplemental brief, which is a classic assistant behavior where the AI is used to augment human writing and research capabilities but requires human verification.
Limited Risk: The AI tools used were general-purpose text generators/chatbots used for drafting and research, which fall under limited risk requiring transparency about AI-generated content.
The AI was used to generate an outline and conduct legal research for a court brief, which aligns with the functions of a writing assistant and question answering system.
The reports are highly detailed court orders and journalistic accounts describing the exact sequence of events, the specific number of fake citations (9 out of 27), the exact financial sanctions ($31,100), and the specific roles of the attorneys involved. There is very little ambiguity about what transpired.
None. The facts of the AI usage and subsequent court sanctions are clear and uncontested.
Technology
AI Model Hallucinations
AI tools generated completely fictional legal citations and phony quotations.
Unreliable AI Research Outputs
Advanced AI models struggled to accurately handle basic case research tasks.
Human Factors
Over-reliance on AI Tools
Attorney outsourced research to AI without verifying the accuracy of output.
Failure to Cite-Check Research
Attorneys at both firms failed to verify citations before filing the brief.
Lack of Disclosure
Attorneys failed to disclose AI use even after being notified of errors.
Process and Methods
Deficient Verification Processes
No formal process was enforced to verify co-counsel's research outputs.
Poor Inter-Firm Communication
K'L Gates incorporated the outline without knowing or asking if AI was used.
Management
Inadequate AI Usage Oversight
Firm management failed to establish guardrails for AI tool adoption.
Prioritization of Speed
Using AI as an unverified shortcut to quickly generate legal briefs.
Regulatory Environment
Lack of Clear AI Use Regulations
Absence of specific court rules governing AI use in legal submissions.
AI system involved
An AI system was involved, but the reports do not identify it.
Identified by automated extraction; can differ from the alleged developers above.
