This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A misinformation expert submitted an affidavit in a Minnesota legal case that included multiple non-existent academic citations. Attorneys for the plaintiffs argued these citations were AI-generated hallucinations from ChatGPT, which has cast doubt on the credibility of the expert's entire declaration and the legal proceedings.
In a legal case defending Minnesota’s deepfake election misinformation law, Stanford misinformation expert Professor Jeff Hancock's affidavit allegedly cited non-existent academic sources, potentially generated by ChatGPT. The reportedly fabricated citations appear to have undermined the credibility of his testimony.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The expert relied on the AI-generated citations without verifying their authenticity, leading to severe reputational and legal consequences.
The AI system exhibited a lack of robustness by hallucinating plausible-sounding but entirely fabricated academic sources.
The AI system generated fabricated academic citations, which constitutes false and misleading information that was subsequently introduced into a federal court proceeding.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The generation of non-existent academic citations was an unintentional hallucination produced by the AI system during its post-deployment use.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
A prominent misinformation expert submitted a federal court declaration containing fabricated academic citations generated by ChatGPT. While highly embarrassing and damaging to the credibility of the legal proceedings regarding deepfake regulations, the incident represents a minor domestic legal disruption with negligible direct national security impact.
The AI system (ChatGPT) was used to assist in drafting or researching the expert declaration, acting as an assistant that requires human verification, which was neglected.
Risk Level 3: Limited Risk. The system is a chatbot or general-purpose AI used to generate text, which falls under limited risk requiring transparency obligations.
The AI was used to generate text and citations for a legal declaration, functioning as a writing and research assistant.
The report clearly details the specific fake citations, the court case, the parties involved, and the consensus that they represent AI hallucinations. The facts are consistent across the reporting.
None
Technology
LLM Citation Hallucination
ChatGPT generated plausible-sounding but entirely fictional academic citations.
Human Factors
Failure to Verify References
The expert signed the declaration without verifying the cited academic papers.
Overreliance on AI Tools
Assuming AI-generated text is accurate without performing manual fact-checks.
Process and Methods
Inadequate Drafting Review
No formal process to cross-reference academic citations before legal submission.
Management
Inadequate Delegation Oversight
Failure to supervise assistants or co-authors who may have used AI tools.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
