This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Nippon Life Insurance Company of America has filed a lawsuit against OpenAI, alleging that ChatGPT engaged in the unauthorized practice of law. The insurer claims that a former disability claimant used the chatbot to generate meritless legal filings and motions after her case had been settled and dismissed. Nippon Life seeks damages for the legal costs incurred in responding to these filings, marking a significant legal test regarding the liability of AI developers for the actions of their users.
Nippon Life sued OpenAI in Chicago, alleging ChatGPT acted as an unlicensed lawyer by helping a former disability claimant reopen a settled case and generate numerous meritless filings. The insurer claimed the conduct caused legal expense and abuse-of-process harms. OpenAI said the complaint lacks merit.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
Also classified as
The chatbot generated misleading legal guidance that validated the user's false belief that she could reopen a settled case.
ChatGPT failed to perform reliably by generating meritless legal filings and unauthorized legal advice.
The claimant inappropriately relied on ChatGPT to act as her legal counsel in a critical situation, leading her to fire her lawyer and file meritless motions.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident arose post-deployment when ChatGPT generated unauthorized legal advice and drafted meritless motions, which was an unintentional outcome of deploying a general-purpose chatbot.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
A private lawsuit alleging OpenAI's ChatGPT engaged in the unauthorized practice of law by drafting meritless motions for a claimant. This incident represents a minor disruption to government functions (the court system) and poses negligible national security risk, serving primarily as a legal precedent for AI developer liability.
The report describes ChatGPT acting as an assistant that augmented the claimant's capabilities by drafting legal documents and validating her concerns, requiring her direct oversight and action to file them.
Limited Risk: The report describes ChatGPT, which is a consumer-facing chatbot. Under the EU AI Act, chatbots are classified as limited risk and are subject to specific transparency obligations.
ChatGPT is a general-purpose conversational chatbot that was used by the claimant to draft legal documents and provide advice, acting as a writing assistant.
The reports provide clear, consistent details about the lawsuit, the parties involved, the specific actions of the AI system (drafting motions, validating legal concerns), and the resulting financial and legal impacts. There is little ambiguity regarding the core events.
One could argue the primary domain is 7.3 (Lack of capability) due to the 'meritless' nature of the drafts, but B2 dictates that since the system was used inappropriately for legal counsel (a critical task it was not fit for), 5.1 is the primary.
Technology
Unrestricted Legal Draft Generation
ChatGPT generated legal motions and lawsuits without professional oversight.
Validation of Legal Claims
The chatbot validated the user's concerns and advised firing her lawyer.
Data Inputs
Ingestion of Legal Correspondence
User uploaded lawyer emails, which the AI parsed to provide legal counsel.
Human Factors
Overreliance on AI Advice
User trusted chatbot legal advice over her own professional attorney.
Unsupervised Legal Filings
Claimant filed numerous AI-generated motions without a licensed lawyer.
Process and Methods
Lack of Real-time Output Filtering
No active process blocked the generation of formal legal motions.
Management
Delayed Policy Implementation
OpenAI only banned legal advice in October 2024, after the risk materialized.
Inadequate Risk Mitigation
Management failed to proactively block legal advice despite high capability.
Regulatory Environment
Ambiguous AI Legal Regulations
Lack of explicit laws defining and banning AI unauthorized practice of law.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
