This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A court-appointed attorney used Westlaw CoCounsel to draft appellate briefs for a criminal defendant, which contained AI-generated false legal quotations and misrepresented case law. The court identified these errors, which necessitated the appointment of new counsel and delayed the appeal. The attorney admitted to the improper use of the AI tool and now faces disciplinary proceedings and loss of compensation.
In United States v. Farris, court-appointed attorney Steven N. Howe admitted using Westlaw CoCounsel to draft appellate briefs that reportedly included AI-generated false quotations and misleading descriptions of precedent. The filing disrupted Farris's appeal, leading the court to appoint new counsel and restart briefing, while Howe lost his compensation and faced possible disciplinary action.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The attorney reflexively relied on the AI-generated draft without verifying the accuracy of the citations, failing to meet his ethical obligations of oversight.
The AI system failed to perform reliably by hallucinating non-existent quotations and misrepresenting actual case holdings.
The AI system generated false legal citations and fabricated direct quotations, which constitutes discrete false or misleading information that misled the attorney and was submitted to the court.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was initiated by the AI system generating hallucinated legal citations and false quotations, which was an unintentional outcome of its text generation process.
An attorney submitted appellate briefs containing AI-generated false legal citations to a federal court. While this disrupted judicial resources and delayed a criminal appeal, it was managed through standard court procedures and carries negligible national security implications.
The AI system, Westlaw CoCounsel, was used to generate a first draft of legal briefs based on uploaded documents, which the attorney then edited, acting as an assistant to augment human writing.
Limited Risk: The AI system is used as a text generator and legal assistant, which falls under Limited Risk (Level 3) as it generates content and requires transparency regarding its AI nature.
The AI system, Westlaw CoCounsel, was used by the attorney's staff to generate a first draft of an appellate brief, acting as a writing assistant for legal document preparation.
The report is a formal judicial opinion from the Sixth Circuit Court of Appeals, providing clear, verified facts about the attorney's actions, the specific AI-generated errors, and the resulting consequences. There is no conflicting information, and the attorney admitted to the facts.
None
Technology
AI hallucination of quotes
CoCounsel generated fake direct quotes and mischaracterized legal holdings.
Misleading citation generation
The tool associated real citations with completely fabricated text.
Data Inputs
Raw court document uploads
Staff uploaded district court documents directly without pre-filtering.
Human Factors
Overreliance on AI tool
Attorney trusted the AI output without performing necessary verification.
Lack of AI familiarity
Attorney used the tool for the first time without understanding its limits.
Inadequate staff supervision
Relying on non-attorney staff to run AI queries without direct oversight.
Process and Methods
Lack of verification process
No systematic process was in place to verify AI-generated legal citations.
Flawed drafting process
Relying on AI for initial drafts without subsequent manual quote checking.
Management
Lack of firm AI policies
The law office lacked policies governing the use and validation of AI tools.
Regulatory Environment
Lack of clear AI guidelines
Absence of strict court rules or specific guidelines on AI use in briefs.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
