This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Two UK court cases revealed that lawyers and clients submitted dozens of completely fabricated legal citations and quotes generated by AI tools, leading to wasted costs orders, reduced costs recovery, and regulatory referrals.
In a London housing case, England's High Court found that a barrister cited five reportedly fictitious cases and ordered her and the Haringey Law Centre, a legal aid charity, to pay £4,000 in wasted costs. Opposing counsel alleged AI use. The barrister denied using generative AI. The Divisional Court identified AI use as one of two possible explanations for the citations.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The lawyers and clients placed inappropriate trust and reliance on generative AI tools to conduct critical legal research without verifying the outputs against authoritative sources.
The AI systems generated highly plausible but completely false legal citations and fabricated quotes, which misled users and were subsequently presented as genuine evidence in court proceedings.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was initiated by generative AI systems hallucinating and outputting completely fictitious legal citations and quotes, which was an unexpected outcome of using the models for legal research.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
Two UK court cases involved lawyers submitting AI-generated fabricated legal citations. While this undermined the administration of justice locally, it was resolved through standard judicial and regulatory processes, posing negligible threat to broader national security.
The reports describe lawyers and clients using generative AI tools to assist with legal research and drafting submissions. The AI acted as an assistant by generating summaries and citations, which required human verification and oversight before being submitted to the court.
Limited Risk: The AI systems used were general-purpose generative AI chatbots and search assistants. According to the definitions, chatbots and general AI-generated content systems pose a moderate risk and fall under Risk Level 3 (Limited Risk).
The AI was used by the lay client and potentially the barrister as a writing and research assistant to generate legal arguments, citations, and summaries for court submissions.
The reports provide highly detailed, official court judgments and legal news articles detailing the exact nature of the fake citations, the parties involved, the financial penalties, and the admissions of AI use by the lay client and suspected use by the barrister.
One could interpret the barrister's actions in the Ayinde case as purely human negligence without AI involvement, but the court found it highly probable that AI was the source of the highly specific, formulaic fake cases.
Technology
Generative AI Hallucinations
AI tools generated highly plausible but entirely fictitious case citations.
Unreliable Search Summaries
Google or Safari AI summaries produced fake cases without user realizing.
Data Inputs
Lack of Authoritative Data
Lawyers failed to cross-reference citations with authoritative sources.
Human Factors
Overreliance on AI Output
Lawyers and clients trusted AI-generated citations without verification.
Incompetence and Negligence
Junior or unqualified staff handled complex legal filings without guidance.
Lack of Professional Diligence
Failure to check citations even after opposing counsel flagged them as fake.
Process and Methods
Absence of Verification Step
No process existed to cross-check citations against official law databases.
Inadequate Supervision
Pupil barrister written work was not checked by chambers supervisors.
Deficient Correspondence Review
Firms dismissed opposing warnings as cosmetic errors without verifying.
Management
Resource Constraints
Overstretched law centre had high caseloads and limited funding.
Delegation to Lay Client
Solicitor relied on lay client to conduct and supply legal research.
Poor Chambers Oversight
Chambers failed to supervise pupil barrister during busy practice.
Regulatory Environment
Insufficient AI Guidance
Guidance promulgated by regulators was insufficient to prevent misuse.
