This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A Colorado attorney used ChatGPT to research and draft a legal motion, relying on the AI to provide case law citations. The model hallucinated several non-existent cases, which the attorney submitted to the court without verification. This led to the denial of the motion, a formal complaint, the attorney's termination from his firm, and a subsequent one-year suspension for violating professional conduct rules.
A Colorado Springs attorney, Zachariah Crabill, mistakenly used hallucinated ChatGPT-generated legal cases in court documents. The AI software provided false case citations, leading to the denial of a motion and legal repercussions for Crabill, highlighting risks in using AI for legal research.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The attorney over-relied on ChatGPT, failing to verify its output before submitting it to the court.
The AI system generated fabricated legal citations (false information) which the attorney relied upon, leading to legal and professional harms.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was initiated by ChatGPT generating fabricated legal citations, which was an unexpected hallucination during a post-deployment search task.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
A Colorado attorney's reliance on ChatGPT resulted in the submission of fabricated legal citations to a state court. While the incident highlights professional risks and reliability issues of generative AI, it has negligible implications for national security.
The report describes ChatGPT acting as an assistant to augment the attorney's capabilities by helping him research and draft a legal motion, requiring human oversight and validation which the attorney failed to perform.
Limited Risk: The system involved is ChatGPT, which is a general-purpose chatbot. According to the definitions, chatbots fall under Risk Level 3 (Limited Risk) and require specific transparency obligations.
The attorney used ChatGPT to help draft a legal motion and search for case law, which aligns with the functions of a writing assistant and a question answering system.
The reports provide consistent, detailed accounts of the incident from multiple sources, including direct quotes from the attorney involved, court documents, and the disciplinary ruling. The facts regarding the AI's role, the specific fabricated citations, and the resulting professional consequences are clear and uncontradicted.
None
Technology
AI Hallucination
ChatGPT generated realistic-looking but completely fabricated legal citations.
Lack of Fact-Checking Mechanism
The AI tool did not verify the existence of the generated citations.
Data Inputs
Lack of Real-Time Legal Data
The model was trained on general text rather than verified legal databases.
Human Factors
Automation Bias and Trust
The attorney assumed the AI's output was correct due to prior accurate answers.
Workplace Stress and Burnout
The attorney was overwhelmed by deadlines and a heavy workload.
Lack of Research Experience
As a rookie lawyer, he lacked experience in drafting and verifying motions.
Process and Methods
Failure to Verify Citations
The attorney did not check the cases in LexisNexis before filing.
Lack of AI Usage Guidelines
No established procedures existed for using generative AI in legal work.
Management
Inadequate Supervision
The law firm failed to review the rookie attorney's work before submission.
High Workload Pressure
Management assigned additional work to an already overwhelmed associate.
Regulatory Environment
Absence of AI Court Rules
Courts lacked clear rules or disclosure mandates regarding generative AI use.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
