This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
In two separate South African court cases, legal practitioners submitted fictitious case citations that were generated by AI tools. The courts identified these 'hallucinations' during proceedings, leading to the dismissal of applications, orders for the law firms to pay additional costs, and mandatory referrals to the Legal Practice Council for professional misconduct. The judgments emphasize that time pressure and the use of new technology do not excuse a failure to verify legal authorities.
In Mavundla v. MEC: Department of Co-Operative Government and Traditional Affairs KwaZulu-Natal, the law firm Surendra Singh and Associates, representing South African politician Philani Godfrey Mavundla, reportedly submitted false legal citations, apparently generated by AI, in filings at the Pietermaritzburg High Court. The court reportedly found that many of the cited cases did not exist in any databases, and the firm's legal team and article clerk allegedly failed to verify the references.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The legal practitioners over-relied on the AI-generated research and failed to perform the necessary professional verification before submitting the documents to court.
The AI system exhibited a lack of robustness by hallucinating realistic-looking but entirely non-existent legal citations when queried for South African case law.
The AI system generated completely fictitious legal citations and case details, which were then submitted to the court as factual authorities.
Human
Due to a decision or action made by humans
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by human legal practitioners failing to verify AI-generated outputs before submitting them to court, which was an unintentional oversight during post-deployment use of the technology.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
Legal practitioners in South Africa submitted AI-generated fictitious case citations in court. While this temporarily disrupted judicial proceedings and highlighted risks of overreliance on AI in legal research, the incident represents negligible national security risk and was handled through standard judicial disciplinary processes.
The AI system acted as an assistant to augment human capabilities in legal research and drafting, but required human oversight to verify its outputs. The report notes that junior counsel used an AI platform 'exclusively trained on South African legal judgments' to assist with drafting heads of argument.
Limited Risk: The AI systems used were general-purpose chatbots and text generation tools (ChatGPT and a specialized legal AI assistant) which fall under transparency obligations for interactive AI and synthetic content generation.
The AI was primarily used as a writing and research assistant to draft legal heads of argument and retrieve case citations.
The reports provide highly detailed, consistent accounts of both court cases, including specific case names, the names of the legal practitioners involved, the judges' rulings, and the exact nature of the AI hallucinations. The role of AI is explicitly confirmed by the practitioners in the Northbound case and strongly established via judicial investigation in the Mavundla case.
None. The facts clearly point to professional negligence involving AI-generated hallucinations.
Technology
AI Hallucinations
AI tools generate coherent but entirely fictitious legal case citations.
Unreliable AI Confirmatory Checks
AI chatbots falsely confirm the existence of non-existent cases.
Data Inputs
Unverified Reference Databases
AI models trained on legal data still produce fabricated citations.
Human Factors
Lack of Professional Vigilance
Practitioners failed to verify AI outputs against authoritative sources.
Dishonesty Regarding AI Use
Staff denied using AI tools when questioned about fictitious cases.
Overreliance on AI Tools
Assuming AI-generated research is accurate without critical evaluation.
Process and Methods
Inadequate Verification Protocols
Lack of systematic cross-checking of citations before court submission.
Time Pressure and Urgency
Severe time constraints and overbooking led to rushed drafting processes.
Deficient AI Risk Training
Lack of training on AI limitations, risks, and ethical implications.
Management
Failure of Staff Supervision
Senior professionals failed to properly supervise candidate attorneys' work.
Inadequate Quality Control
Firms lacked rigorous internal review mechanisms for court submissions.
Resource Constraints
Firms lacked access to or refused to pay for official law databases.
Regulatory Environment
Evolving Legal Standards
Absence of clear, preemptive guidelines on AI usage in court filings.
Mandatory Judicial Referrals
Code of Judicial Conduct obliges judges to report professional misconduct.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
