This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A lawyer at Binnall Law Group used Anthropic's Claude Console to draft a court motion, which resulted in the inclusion of non-existent 'phantom' legal quotations. The attorney admitted to failing to verify the AI-generated content and apologized to the federal judge. The firm has since committed to implementing additional safeguards and training regarding the use of AI in legal practice.
A Binnall Law Group attorney reportedly used Anthropic's Claude Console to draft a motion to quash a subpoena in litigation over Trump administration layoffs, then filed a version containing "phantom" quotations that did not appear in the cited cases. The attorney apologized to U.S. District Judge Susan Illston, and the firm said it would add safeguards and training after calling the AI-generated errors unacceptable.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The attorney relied on the AI-generated draft for a critical legal filing without verifying the accuracy of the citations.
The Claude AI system generated false and misleading legal citations ('phantom' quotations) that were subsequently submitted in a federal court filing.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the Claude AI system generating hallucinated legal citations during its post-deployment use by an attorney.
An attorney at Binnall Law Group submitted a federal court filing containing three AI-generated 'phantom' citations after failing to verify a draft produced by Anthropic's Claude. The incident represents a minor, localized disruption to judicial proceedings that was quickly resolved through standard court oversight, carrying negligible broader national security implications.
The attorney used Claude Console to produce an initial draft of a court filing, which is an assistant behavior requiring human oversight and verification.
Risk Level 3: Limited Risk. The system used was Anthropic's Claude Console, which functions as a chatbot and text generation tool, falling under the category of AI-generated content.
The AI system was used by the attorney to draft a legal motion, which is a writing assistant function.
The reports provide clear, direct, and uncontradicted details about the incident, including the specific AI tool used (Claude Console), the law firm involved (Binnall Law Group), the lawyer's admission of responsibility, and the nature of the AI's failure (generating three phantom quotations).
None
Technology
AI Generation of Phantom Quotes
Claude Console generated non-existent legal citations in the draft.
Lack of Built-in Fact Checking
The AI tool lacked automated verification against real legal databases.
Data Inputs
Unverified Draft Text
The initial text input to the filing contained unverified AI outputs.
Human Factors
Failure to Vet AI Results
The attorney failed to adequately review and verify the AI-generated text.
Tight Time Constraints
Severe time pressure contributed to the failure to properly review the filing.
Overreliance on AI Output
The attorney trusted the AI's generated citations without verification.
Process and Methods
Bypassing Verification Policies
Firm policies requiring independent verification of AI content were bypassed.
Inadequate Drafting Safeguards
The workflow did not enforce cross-referencing AI text with legal databases.
Management
Inadequate Policy Enforcement
Management failed to ensure adherence to policies restricting AI tool use.
Lack of Supplemental Training
Attorneys lacked sufficient training on risks associated with generative AI.
Regulatory Environment
Lack of Strict AI Court Rules
Existing court rules rely on retroactive sanctions rather than prevention.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
