This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Legal practitioners in South Africa used an AI tool called 'Legal Genius' to draft court submissions for an urgent licensing dispute. The tool generated fictitious case law, which was included in the heads of argument filed with the Johannesburg High Court. The presiding judge discovered the fabrications, and despite the legal team's apologies and lack of intent to deceive, the court referred the practitioners to the Legal Practice Council for investigation, citing the serious risks AI-generated 'hallucinations' pose to the administration of justice.
In South Africa, the legal team for Northbound Processing reportedly admits that non-existent case citations in an urgent licensing dispute with the South African Diamond and Precious Metals Regulator were purportedly produced by the Legal Genius AI tool. The Johannesburg High Court found the fake references undermined the credibility of Northbound's submissions and referred the lawyers' conduct to the Legal Practice Council for investigation.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The legal practitioners overrelied on the AI tool under time pressure without verifying the citations against authoritative sources.
The AI system generated fictitious legal citations in a court brief, which constitutes discrete false or misleading information as outlined in the boundary rules.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the AI system 'Legal Genius' generating fictitious citations post-deployment, which was an unexpected and unintentional outcome of its use for legal research.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
Legal practitioners in South Africa unintentionally submitted AI-generated fictitious case law to the Johannesburg High Court. While the incident highlights risks to judicial integrity and public trust in the legal system, it carries negligible national security implications and was resolved through standard judicial oversight and professional referral.
The AI system 'Legal Genius' acted as an assistant by generating legal research and drafting suggestions for the junior counsel, who was expected to oversee and verify its outputs before submitting them to the court.
Limited Risk: The AI system 'Legal Genius' is a generative AI tool used to produce AI-generated content (legal citations and arguments), which requires transparency and verification to prevent misleading the court.
The AI system 'Legal Genius' was used by legal practitioners as a writing and research assistant to draft heads of argument for a court case.
The reports provide consistent, detailed accounts of the specific legal case, the names of the parties and legal practitioners involved, the specific AI tool used ('Legal Genius'), and the exact nature of the AI's failure (hallucinating citations). There are no conflicting accounts of the core events.
None
Technology
AI Hallucination
The GenAI tool generated fictitious case citations that did not exist.
Misleading Tool Claims
Tool marketed as trained solely on South African law but still hallucinated.
Data Inputs
Unverified Training Outputs
The model produced plausible but entirely fabricated legal reference citations.
Human Factors
Failure to Verify Citations
Junior counsel relied on AI outputs without checking authoritative sources.
Superficial Review by Senior Counsel
Senior counsel performed only a sense-check instead of full verification.
Lack of AI Risk Understanding
Practitioners used AI without understanding hallucination risks.
Process and Methods
Inadequate Verification Protocol
No structured workflow existed to cross-reference AI outputs with real cases.
Severe Time Constraints
Urgent nature of the application limited time for citation verification.
Drafting Process Disruption
The sudden withdrawal of the original junior counsel disrupted the workflow.
Management
Inadequate Professional Oversight
Lack of supervisory control to ensure compliance with professional duties.
Regulatory Environment
Mandatory Judicial Referral
Code of Judicial Conduct Article 16(1) obliges reporting of misconduct.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
