This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A Stanford University study found that specialized legal AI tools from LexisNexis and Thomson Reuters, as well as general-purpose models like GPT-4, frequently hallucinate, providing incorrect legal information and fake citations. These errors pose significant risks to legal professionals and litigants who may rely on the tools for authoritative research. The study highlights a lack of transparency and rigorous benchmarking in the legal AI industry.
Stanford University’s Human-Centered AI Institute (HAI) conducted a study in which they designed a "pre-registered dataset of over 200 open-ended legal queries" to test AI products by LexisNexis (creator of Lexis+ AI) and Thomson Reuters (creator of Westlaw AI-Assisted Research and Ask Practical Law AI). The researchers found that these legal models hallucinate in 1 out of 6 (or more) benchmarking queries.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The AI systems generate false legal precedents and fabricated citations, leading to inaccurate beliefs in users.
Lawyers and litigants place inappropriate trust in the AI outputs without verification, leading to legal sanctions and court errors.
The AI systems fail to perform reliably under varying conditions, generating hallucinations and incorrect legal citations in a critical domain.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The hallucinations and incorrect legal citations are unintended failures of the AI systems occurring after they have been deployed to users.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
A Stanford study revealed that legal AI tools frequently hallucinate, leading to fabricated citations being introduced into court proceedings. While representing a minor concern for judicial administration and access to justice, the national security implications are negligible to minor, with no threat to critical infrastructure, intelligence, or state sovereignty.
The AI systems function as assistants designed to augment human capabilities by providing legal information, suggestions, and drafting support, but they require human oversight and validation.
Limited Risk: The report describes the systems as 'AI chatbots' and generative AI tools used for legal queries, which fall under the category of chatbots requiring transparency obligations.
The AI systems are primarily used to answer complex legal queries and retrieve relevant case law, acting as question-answering systems for legal research.
The reports are highly detailed, citing specific academic studies (Stanford RegLab/HAI and Open University) and real-world legal cases (Manchester and New York). The findings are consistent across both reports.
None
Technology
Inherent LLM Hallucinations
Generative AI models naturally tend to invent false legal information.
Retrieval Mechanism Failures
RAG systems fail to retrieve the correct or best legal authorities.
Model Sycophancy
AI models tend to agree with the user's incorrect assumptions.
Data Inputs
Out-of-Date Training Data
Models lack recent legal developments, referencing replaced laws.
Inapplicable Authority Retrieval
Systems retrieve semantically similar but legally irrelevant sources.
US-Centric Training Bias
Chatbots default to US law without specifying jurisdiction differences.
Human Factors
Overreliance on Well-Written Text
Users assume professional-looking AI outputs are correct and authoritative.
Lack of User Legal Expertise
Pro se litigants cannot verify if AI-generated citations are real.
Mistaken User Assumptions
Users input false premises which the AI naively validates.
Process and Methods
Lack of Rigorous Benchmarking
Industry lacks public evaluations and hard metrics to assess reliability.
Opaque AI Tool Design
Providers publish few details about model design and performance metrics.
Inadequate Verification Processes
Users fail to verify every AI-generated proposition and citation.
Management
Opaque Performance Claims
Management markets tools as 'hallucination-free' without providing evidence.
Neglecting Independent Evaluation
Firms adopt AI without developing hard metrics to measure actual accuracy.
Inefficient Evaluation Methods
Firms spend excessive time testing tools without clear efficiency metrics.
Regulatory Environment
Lagging Supervision Guidelines
Regulatory bodies only recently issued guidance on AI supervision duties.
Lack of Standardized Benchmarks
No regulatory mandates exist for transparent reporting of AI error rates.
Inconsistent Court Standing Orders
Judges independently issue varying disclosure rules for courtroom AI use.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
