This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
In a defamation case at the Johannesburg Regional Court, a legal team relied on fictitious case law generated by ChatGPT. The magistrate discovered the citations were non-existent, leading to a two-month delay in the proceedings and a punitive costs order against the plaintiff's attorneys for failing to verify the AI-generated research.
In a defamation case at the Johannesburg Regional Court, Rodrigues Blignaut Attorneys, representing plaintiff Michelle Parker, reportedly relied on purportedly non-existent legal judgments generated by ChatGPT to help argue their case. Magistrate Arvin Chaitram reportedly found the case names and citations were fictitious, causing a two-month delay. The court issued a punitive costs order and rebuked the plaintiff's legal team for uncritically accepting AI-generated research.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The lawyers overrelied on the chatbot's outputs for critical legal research without verifying them.
The incident involves ChatGPT generating fabricated legal citations, which falls under false or misleading information.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by ChatGPT generating fictitious legal citations (AI entity) unintentionally during post-deployment use.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
A South African legal team presented fictitious, ChatGPT-generated case citations in a defamation lawsuit, leading to a two-month postponement and punitive costs. This incident highlights the professional risks of generative AI hallucinations and human overreliance, but carries negligible national security implications.
The report describes ChatGPT being used as a research assistant to find legal precedents. The lawyers used the tool to supplement their work, but failed to provide the necessary human oversight and verification required for assistant-type AI systems.
Limited Risk: ChatGPT is a chatbot, which is explicitly classified as a limited risk system requiring transparency under the EU AI Act.
The AI system was used by lawyers as a writing and research assistant to find legal precedents and draft arguments.
The reports provide consistent, detailed accounts of the South African court incident, including quotes from the magistrate's judgment, details of how the fabrication was discovered, and the consequences (punitive costs and delay). The facts are well-corroborated across multiple news sources.
None.
Technology
AI Hallucination
ChatGPT generated plausible but entirely fictitious legal cases and citations.
Lack of Real-World Understanding
The LLM lacks actual comprehension of legal contexts and factual reality.
Data Inputs
Training Data Limitations
The AI model was trained on datasets that limit its accuracy in niche contexts.
Human Factors
Overzealous and Careless Use
Lawyers accepted AI outputs without verifying the accuracy of the citations.
Uncritical Belief in AI
Attorneys relied blindly on ChatGPT to supplement critical legal research.
Lack of Technical Familiarity
Lawyers used advanced AI tools without understanding their operational limits.
Process and Methods
Lack of Independent Verification
Attorneys failed to cross-verify the AI-generated cases with official sources.
Absence of Human-in-the-Loop
No robust review process was used to validate the AI outputs before sharing.
Management
Inadequate AI Training
Law firms failed to train staff on the proper and safe utilization of AI.
Lack of Verification Policies
Management did not enforce mandatory verification steps for AI-sourced data.
Regulatory Environment
Lack of AI Legal Standards
Absence of clear regulatory standards on using generative AI in legal briefs.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
