This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
An attorney was sanctioned by the 10th Circuit Court of Appeals after using ChatGPT to generate legal research for an appellate brief. The AI produced seven non-existent cases and fabricated quotes, which the attorney failed to verify before filing. The court ordered the attorney to pay $1,000 in sanctions to the opposing party and referred the matter to disciplinary authorities.
In Amarsingh v. Frontier Airlines, the Tenth Circuit sanctioned attorney Kusmin L. Amarsingh after finding that her appellate brief included fabricated case citations and misattributed legal authority generated by ChatGPT and not verified before filing. The court ordered a $1,000 payment to Frontier and referred the matter to Maryland disciplinary authorities.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The attorney overrelied on ChatGPT for legal research and failed to perform the fundamental duty of verifying the accuracy of the generated citations.
The AI system generated fabricated legal citations and quotes (false information) which misled the attorney into submitting them to a federal court, resulting in financial sanctions.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was initiated by the AI system generating fabricated legal citations and quotes, which is an unintentional hallucination outcome of the model's generation process.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
An attorney was sanctioned $1,000 by a U.S. federal appeals court for submitting a brief containing fabricated citations generated by ChatGPT. The incident represents a case of professional negligence and AI overreliance with negligible implications for national security.
The attorney used ChatGPT as a writing and research assistant to help draft her legal brief, relying on its outputs without independent verification.
Risk Level 3. Limited Risk: The system involved is ChatGPT, a general-purpose chatbot and generative AI tool. The report describes the attorney's 'reliance on generative AI tools, specifically ChatGPT, to assist in legal research and drafting'.
The attorney explicitly used ChatGPT to assist in legal research and drafting, requesting specific case law to support her arguments.
The report provides clear, unambiguous details about the incident, including the specific court order, the attorney's admission, the exact number of fake citations, and the financial penalty. There are no conflicting accounts.
None
Technology
Generative AI Hallucination
ChatGPT generated plausible-sounding but completely fictitious legal case names.
Non-Existent Case Quotation
The AI tool fabricated quotes and attributed them to non-existent cases.
Plausible Output Formatting
AI output matched standard legal citation styles, masking its inaccuracy.
Data Inputs
Lack of Real-Time Case Validation
The AI model lacked integration with live, verified legal databases.
Human Factors
Overreliance on GenAI Tools
The lawyer assumed ChatGPT output was accurate without independent verification.
Failure to Cross-Reference
The attorney failed to check the generated citations against actual case texts.
Misunderstanding of AI Sophistication
The attorney believed the tool's sophistication guaranteed factual accuracy.
Process and Methods
Inadequate Legal Brief Review
The process of drafting lacked a mandatory verification step for citations.
Careless Use of AI in Research
The drafting process relied solely on AI output for critical legal precedents.
Management
Neglect of Professional Duty
The lawyer failed to manage and verify her own legal work product.
Regulatory Environment
Evolving Professional Rules
Rules warning against improper AI use were only recently being amended.
Lack of Pre-Filing Warnings
Courts were only beginning to explicitly caution lawyers about AI pitfalls.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
