This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Two U.S. federal judges admitted that staff in their chambers used ChatGPT and Perplexity to draft court rulings that were subsequently docketed despite containing fabricated legal citations, quotes, and parties. The errors were identified by litigants, leading to the retraction of the documents and prompting a Senate Judiciary Committee investigation into the oversight of AI use in the judiciary.
Two U.S. federal judges, Julien Neals (D.N.J.) and Henry Wingate (S.D. Miss.), reportedly admitted that staff in their chambers used AI tools, ChatGPT and Perplexity, to draft rulings containing purportedly fabricated quotes, cases, and parties. The erroneous filings were reportedly docketed before review, prompting Senate and judicial inquiries. Both rulings were retracted.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
Court staff overrelied on generative AI tools for legal research and drafting without verifying the outputs, leading to the docketing of erroneous rulings.
The AI models failed to perform reliably by hallucinating non-existent legal precedents and facts when applied to legal drafting tasks.
The AI systems inadvertently generated fabricated legal citations, quotes, and case details, which were then published in official court dockets, spreading false information within the judicial system.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by ChatGPT and Perplexity generating hallucinated legal citations and quotes, which was an unintentional and unexpected outcome during post-deployment use by court staff.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
U.S. federal court staff used ChatGPT and Perplexity to draft rulings, resulting in the docketing of documents with hallucinated legal citations. While this compromised judicial decision-making processes, the national security impact is minor and is being managed through administrative guidance and internal reviews.
The report describes the AI systems being used as a 'foundational drafting assistant' and to 'perform legal research' to augment human capabilities. These systems required human oversight and verification, which failed in these instances, fitting the 'Assistant' definition.
Limited Risk: The tools used were general-purpose chatbots (ChatGPT and Perplexity) which are subject to transparency obligations. The report notes staff used 'ChatGPT to perform legal research' and 'Perplexity as a drafting assistant'.
The AI systems were used by court staff as writing assistants to draft legal documents and perform legal research for judicial decisions.
The reports are highly detailed, coming from reputable news sources, and contain direct admissions from the federal judges involved, along with specific details of the cases and the AI tools used.
None
Technology
Generative AI Hallucinations
AI tools confidently invented facts, fake quotes, and nonexistent cases.
Use of Unverified AI Assistants
Perplexity and ChatGPT were used for drafting and legal research.
Human Factors
Overreliance on AI Tools
Staff relied on AI output without independently verifying the cited cases.
Lack of Staff Training
Clerk and intern lacked understanding of generative AI hallucination risks.
Negligent Verification
Staff failed to read and verify the actual cases cited by the AI.
Process and Methods
Inadequate Review Processes
Draft opinions were docketed without undergoing a final review process.
Lack of Written Chamber Policies
Chambers relied on verbal understandings rather than written AI bans or rules.
Premature Document Docketing
Drafts containing AI errors were mistakenly published before final approval.
Management
Inadequate Judicial Oversight
Judges failed to adequately scrutinize documents issued under their names.
Lack of Initial Transparency
Judges did not immediately disclose AI use when errors were first flagged.
Busy Docket Management Pressures
High workload pressured chambers to use AI tools as drafting assistants.
Regulatory Environment
Absence of Binding AI Rules
Federal courts lacked formalized, decisive measures to prevent AI misuse.
Insufficient Interim Guidance
Interim guidance only suggested considering AI disclosure, lacking mandates.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
