This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A South African litigant submitted legal pleadings containing completely fabricated case citations and a non-existent judgment generated by an artificial intelligence system.
Self-represented litigant Adriaan Johannes Roux relied on a nonexistent judgment in High Court submissions in Johannesburg, South Africa. The court attributed the false authority and unsound legal arguments to his admitted AI use and said addressing the material wasted court time and resources. The specific AI tools were not identified.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The litigant uncritically trusted and relied on the AI's outputs without performing necessary verification, leading to severe legal and financial consequences.
The AI system generated discrete false information in the form of fabricated legal citations and a non-existent court judgment, which were then presented as factual in court.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the AI system generating unintended hallucinations (fabricated citations) during post-deployment use by a litigant.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
A South African litigant submitted AI-generated fabricated case citations to the High Court. The court identified the hallucinations and dismissed the case with punitive costs. The incident is a domestic civil matter managed through standard judicial procedures, presenting negligible national security implications.
The AI system was used to augment the litigant's capabilities by providing research support and drafting assistance, but required human oversight and validation which was neglected.
Limited Risk: The incident involves a Large Language Model used as a writing assistant and chatbot, which falls under generative AI systems subject to transparency obligations.
The AI was used by the litigant as a research and drafting assistant to generate legal submissions and arguments.
The court judgment provides detailed, first-hand factual findings regarding the litigant's use of AI, the specific fabricated citations, and the resulting legal consequences. There is no ambiguity about the AI's role or the outcomes.
None
Technology
AI Hallucinations
LLMs generated highly convincing but entirely fabricated legal case citations.
Predictive Statistical Output
LLMs generate responses based on statistical patterns, not legal reasoning.
Lack of Source-Linked Outputs
General purpose AI lacks verified databases and audit trails for citations.
Data Inputs
Unverified Training Data
AI model was trained on general data rather than a controlled legal database.
Human Factors
Uncritical Reliance on AI
The practitioner accepted AI outputs without any human verification.
Cognitive and Physical Stress
Litigant used AI to cope with hearing impairment and cognitive difficulties.
Distrust of Legal Professionals
Litigant's belief that lawyers cannot be trusted led to self-use of AI.
Process and Methods
Lack of Human Oversight
No process was in place to verify the accuracy of AI-generated submissions.
Bypassing Procedural Remedies
Litigant used AI to draft urgent applications instead of standard procedures.
Failure to Correct Errors
Doubling down on hallucinated cases instead of verifying court registries.
Management
Failure of Professional Duties
Negligent use of AI violated core duty to confirm authority accuracy.
Regulatory Environment
Insufficient AI Guidelines
Lack of specific court directives on the permissible use of generative AI.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
