This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Michael Cohen, a former lawyer, used Google Bard to generate legal citations for a motion to end his supervised release. He mistakenly believed the AI was a search engine and provided the resulting fake citations to his attorney, David M. Schwartz, who included them in a court filing without verification. The presiding judge discovered the cases did not exist and requested an explanation, leading to public admission of the error and a subsequent ruling that declined to impose sanctions, citing a lack of bad faith.
Michael Cohen, former lawyer for Donald Trump, claims to have used Google Bard, an AI chatbot, to generate legal case citations. These false citations were unknowingly included in a court motion by Cohen's attorney, David M. Schwartz. The AI's misuse highlights emerging risks in legal technology, as AI-generated content increasingly infiltrates professional domains.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The lawyer and client overrelied on the AI system's outputs, submitting them to a federal court without verifying their authenticity.
The AI system generated fabricated legal citations that were presented as factual in a court filing, misleading the users and the court.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk arose from the AI system generating fabricated legal citations (hallucinations) during active use by an end-user.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
A former lawyer and his attorney unintentionally submitted fabricated AI-generated legal citations to a U.S. federal court. While highlighting the risks of professional overreliance on generative AI and causing minor administrative disruption to judicial proceedings, the incident has negligible national security implications.
The AI system acted as an assistant to help with legal research, generating text and citations that were meant to be reviewed and verified by a human before being used in a court filing.
Risk Level 3: Limited Risk. The system used is a general-purpose chatbot (Google Bard/ChatGPT), which falls under the category of chatbots and generative AI systems that require transparency but do not pose unacceptable or high risks by default.
The AI system was used to generate text, citations, and summaries to assist in drafting a legal motion.
The reports provide consistent, detailed, and unsealed court-documented accounts of the incident, including direct quotes from the judge, Michael Cohen, and his attorney. The facts of the AI's hallucination and its subsequent submission to the court are clear and undisputed.
None
Technology
AI Hallucinations
Google Bard generated realistic but completely fictional court citations.
High Error Rates in LLMs
LLMs produce false legal info between 69% and 88% of the time.
Overconfident AI Outputs
Models fail to qualify answers with appropriate levels of confidence.
Data Inputs
Sequence Prediction Flaws
Models guess text fragments based on training rather than retrieving facts.
Human Factors
User Misunderstanding of AI
Cohen mistook the generative chatbot for a supercharged search engine.
Lack of Legal Tech Awareness
Cohen was unaware of emerging trends and risks in generative AI tools.
Overreliance and Blind Trust
Lawyer submitted client-provided AI citations without verifying them.
Process and Methods
Lack of Human Verification
No process was used to verify the existence of the cited court cases.
Poor Communication in Team
Lawyer falsely assumed citations came from co-counsel rather than client.
Inadequate Quality Control
Law firm lacked standard procedures to check documents before filing.
Management
Failure to Monitor Tech Risks
Legal team failed to establish policies regarding client-provided research.
Regulatory Environment
Lack of AI Legal Standards
Courts are still grappling with how to regulate generative AI in filings.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
