This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The UK's Crown Prosecution Service apologized to the High Court after using generative AI to draft extradition court documents that contained fabricated, non-existent legal citations which went unchecked by reviewing lawyers.
In joined appeals involving Andreea-Maria Tobosaru and Marian Tofan, the Crown Prosecution Service reportedly filed grounds of opposition containing two nonexistent authorities that likely originated from purported generative AI. The citations also appeared in a later document. The High Court said they did not affect the appeals, and the CPS apologized, citing inadequate verification by a reviewing lawyer.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The CPS lawyers relied on the AI-generated document without verifying its accuracy before filing it in court.
The AI system failed to perform reliably by hallucinating non-existent legal authorities during legal research.
The AI system inadvertently generated incorrect and fabricated legal citations, which were then included in formal court submissions.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by an AI system generating non-existent legal citations (hallucinations) during post-deployment use.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
The UK Crown Prosecution Service used generative AI to draft extradition documents containing fabricated legal citations. While this compromised the quality of submissions to a core government function, it was caught by human review and did not result in systemic harm, representing a minor national security impact managed through standard legal procedures.
The AI was used to compose court documents and conduct legal research, acting as an assistant to the CPS lawyers who were responsible for reviewing and verifying its output before filing.
Limited Risk: The system is a generative AI used to compose text, which falls under AI-generated content obligations requiring transparency.
The CPS used the AI system to draft and compose court documents for extradition cases, which is the primary function of a writing assistant.
The report provides clear, direct facts about the incident, including the organization involved (CPS), the nature of the AI error (hallucinated citations), how it was discovered, and the lack of actual harm. There is no conflicting information or major ambiguity regarding what transpired.
None. The facts of the case are straightforward and accepted by all parties involved.
Technology
Generative AI Hallucination
The AI generated non-existent legal citations and authorities.
Human Factors
Lack of Verification
The reviewing lawyer failed to check the accuracy of the document.
Inadequate Checking by Counsel
Junior counsel did not verify authorities before preparing documents.
Process and Methods
Flawed Document Review Process
The CPS process did not catch AI errors before filing and serving.
Management
Inadequate Risk Assessment
CPS lacked proper risk assessment for using generative AI in court work.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
