This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A law firm submitted a court filing containing fabricated case citations and quotations generated by an unchecked legal AI tool, leading to a federal judge issuing a warning about the risks of AI in legal practice.
Law firm Shapiro Arato Bach reportedly acknowledged that unchecked use of Harvey, a legal AI platform, produced a fictitious case citation and misattributed quotations in a brief filed in federal court in New York. The court required an explanation and disclosure of the AI tool. The judge declined sanctions, and the firm apologized and said it was adding safeguards.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The drafting lawyer overrelied on the AI tool's output, failing to perform the standard professional verification of the generated citations.
The legal AI tool exhibited a lack of robustness by hallucinating non-existent case citations and misattributing legal principles.
The AI tool generated fabricated legal citations and false quotations (hallucinations) which were then submitted to a federal court, presenting false information as fact.
Human
Due to a decision or action made by humans
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by a human lawyer failing to verify AI-generated outputs before filing them in court, which was an unintentional oversight contrary to firm policy.
A private law firm submitted a court filing in a civil copyright case containing fabricated legal citations generated by an AI tool. While this represents a failure of professional conduct and highlights risks of unchecked AI use in the legal system, it was an isolated, unintentional error with negligible implications for national security.
The AI tool developed by Harvey was used to help prepare a legal filing, acting as an assistant to augment the lawyer's drafting capabilities while requiring human oversight and validation.
Limited Risk: The system is a generative AI tool used for drafting text (similar to a chatbot or content generator) which requires transparency and human verification, but does not fall under high-risk categories like critical infrastructure or law enforcement.
The AI tool was licensed and used specifically to assist lawyers in drafting legal briefs and responses.
The reports provide highly detailed, first-hand legal declarations and court details explicitly outlining how the AI was used, what errors were generated, and the organizational context of the failure.
None
Technology
Generative AI Hallucination
The Harvey AI tool generated fake citations and quotes that did not exist.
Opaque Pricing Recommendations
AI-driven pricing engine created confusing and wide variations in food prices.
Complex Algorithmic Rules
AI engine parameters and corporate targets created conflicting pricing signals.
Data Inputs
Competitor Price Scraping
Pulling online menu data from competitors introduces legal and antitrust risks.
Granular Transaction Data Analysis
Analyzing millions of daily transactions led to localized price discrimination.
Human Factors
Failure to Verify AI Outputs
The responsible lawyer did not check the AI-generated case citations.
Distraction and Personal Bereavement
The drafting lawyer suffered personal loss, impacting her focus and diligence.
Overreliance on Well-Known Principles
Partner assumed citations were correct because principles were uncontroversial.
Process and Methods
Violation of Firm AI Policy
The drafting attorney acted contrary to the firm's AI validation policies.
Lack of Independent Draft Verification
No secondary process caught the hallucinated citations before court filing.
Aggressive Franchisee Pressure
Corporate enforcement systems penalized franchisees who deviated from AI pricing.
Management
Inadequate Training Enforcement
Mandatory AI ethics training failed to prevent individual compliance errors.
Prioritizing Margin over Compliance
Corporate pushed AI pricing recommendations to optimize top-line revenue.
Lack of Mentorship and Oversight
Insufficient supervision of junior lawyers using generative AI tools.
Regulatory Environment
Antitrust and Collusion Scrutiny
Regulators like the FTC scrutinize algorithms that coordinate competitor prices.
Increasing Judicial Sanctions
Courts are increasingly weighing sanctions for unchecked AI legal submissions.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
