This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A defense lawyer in the Supreme Court of Victoria filed legal submissions containing fictitious quotes and nonexistent case citations generated by an unidentified AI tool. The errors were discovered by the judge's associates, leading to a 24-hour delay in the murder trial. The lawyer apologized to the court, and the judge emphasized that AI outputs must be independently verified to maintain the integrity of the justice system.
Australian defense lawyer Rishi Nathwani KC reportedly filed submissions in a Supreme Court of Victoria murder case that included allegedly fictitious quotes and nonexistent case citations purportedly generated by AI. The errors were reportedly discovered by the judge's associates, causing a 24-hour delay in the proceedings. Nathwani apologized in court, taking full responsibility, while Justice James Elliott criticized the incident and reiterated that AI outputs must be independently verified.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The defense lawyer relied on the AI system's output for critical legal submissions without independent verification.
The AI system generated false and fabricated legal citations and quotes, leading the lawyer to submit incorrect information to the court.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by an AI system generating fabricated quotes and nonexistent case citations, which was an unexpected outcome of using the tool for legal research.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
A defense lawyer in Australia submitted AI-generated fictitious case citations and quotes to the Supreme Court of Victoria, causing a 24-hour delay in a murder trial. The incident represents a minor, localized disruption to judicial operations due to professional negligence and overreliance on AI, with negligible broader national security implications.
The AI system acted as an Assistant, designed to augment the lawyer's legal research and drafting capabilities, but requiring human oversight and verification which was neglected in this case.
Limited Risk: The report describes the use of an unidentified generative AI system, which falls under Risk Level 3 as a general-purpose AI tool capable of generating text and content.
The AI system was used by the defense lawyer to generate legal submissions, acting as a writing and research assistant.
The report provides clear, consistent details about the incident, including the specific court, the lawyer involved, the nature of the AI-generated errors (fabricated quotes and citations), and the resulting 24-hour delay. The lawyer took full responsibility, and the judge's statements are directly quoted.
None. The event is a straightforward case of professional negligence involving AI-generated hallucinations.
Technology
Generative AI Hallucinations
The AI system generated nonexistent court cases and fake legal quotes.
Lack of Source Integration
The AI tool lacked integration with verified legal databases to check facts.
Data Inputs
Unverified Training Data
AI models trained on text patterns without factual verification mechanisms.
Human Factors
Overreliance on AI Output
Lawyers wrongly assumed all generated citations were correct after checking one.
Lack of AI Literacy
Lawyers did not understand that generative AI can hallucinate realistic text.
Process and Methods
Inadequate Verification Process
Defense lawyers failed to independently verify the AI-generated citations.
Lack of Prosecution Cross-Check
The prosecutor did not check the accuracy of the received submissions.
Management
Inadequate Supervision
Senior counsel failed to supervise the use of AI tools within the case.
Regulatory Environment
Unenforced Legal Guidelines
Court guidelines on AI use were not strictly monitored or enforced.
