This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The National Eating Disorders Association (NEDA) implemented a wellness chatbot named Tessa to provide support for individuals with eating disorders. Following the unionization of NEDA's human helpline staff, the organization announced plans to wind down the human-staffed helpline and transition to the chatbot. Shortly after, users and experts reported that Tessa was providing harmful, pro-eating-disorder advice, such as recommending calorie restriction and weight loss techniques. NEDA subsequently took the chatbot offline to investigate, citing that the harmful responses were against their policies and core beliefs.
The National Eating Disorders Association (NEDA) has shut down its chatbot named Tessa after it gave weight-loss advice to users seeking help for eating disorders. The incident has raised concerns about the risks of using chatbots and AI assistants in healthcare settings, particularly in addressing sensitive issues like eating disorders. NEDA is investigating the matter, emphasizing the need for caution and accuracy when utilizing technology to provide mental health support.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
NEDA attempted to transition a critical helpline service to an automated chatbot, representing unsafe use of AI in a sensitive mental health context.
The chatbot failed to perform reliably and safely under user interaction, generating harmful and medically inappropriate advice due to an unapproved generative AI upgrade.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The immediate cause of the risk was the AI chatbot generating unscripted, harmful weight-loss advice to vulnerable users post-deployment, which was an unintentional outcome of its programming.
Added categories peak at 2: Minor (Psychological Harm, Direct)
The NEDA wellness chatbot Tessa was taken offline after an unauthorized generative AI upgrade led it to provide harmful weight-loss advice to users. While representing a significant failure in AI safety and vendor oversight within a sensitive health context, the incident has negligible national security implications.
The Tessa chatbot was deployed as an automated wellness assistant to provide information, coping strategies, and guide users through a structured program, acting to augment NEDA's support services.
Limited Risk: The system is a chatbot designed to interact with users, which falls under the Limited Risk category requiring transparency obligations so users know they are interacting with an AI.
The chatbot was specifically designed and deployed to assist individuals struggling with body image issues and prevent eating disorders, functioning as a health and wellness AI assistant.
Multiple high-quality investigative reports from major outlets (WSJ, NYT, NPR) provide consistent accounts of the chatbot's behavior, the organizational context of NEDA, and statements from both NEDA and the developer Cass. The technical cause (unauthorized generative AI upgrade) is clearly identified.
None
Technology
Unsanctioned Generative AI Upgrade
Cass added generative AI features without NEDA's knowledge or authorization.
Off-Script Response Generation
The chatbot went off-script and generated unapproved weight-loss advice.
Inability to Detect Eating Disorder
The bot failed to recognize user disclosures of active eating disorders.
Data Inputs
Mismatched Training Datasets
Generative models used standard diet advice unsuitable for eating disorders.
Lack of Input Validation for Crisis
The system did not properly filter or flag crisis-related user inputs.
Nefarious Inputs by Bad Actors
Bad actors actively tried to trick the bot during a massive traffic surge.
Human Factors
User Expectation of Empathy
Vulnerable users expected human-like empathy from an automated chatbot.
Lack of Professional Supervision
No human professionals monitored the chatbot interactions in real-time.
Overreliance on Automated Advice
Users trusted the bot's advice despite its inappropriate recommendations.
Process and Methods
Inadequate Integration Testing
The upgraded generative features were not thoroughly audited before deployment.
Lack of Collaborative Review
NEDA and developers failed to co-verify the bot's updated content.
Deficient Safety Guardrails
The system's guardrails failed to block restrictive dieting tips.
Management
Uncoordinated Vendor Management
NEDA did not maintain visibility into Cass's technical updates.
Premature Helpline Closure
Management closed the human helpline before validating the chatbot.
Growth Prioritization Over Safety
NEDA sought to scale prevention tools rapidly without proper risk controls.
Regulatory Environment
Mandated Reporting Liabilities
Complex state and federal reporting rules drove NEDA to replace humans.
Absence of AI Healthcare Standards
Lack of clear regulations for deploying AI in mental health settings.
Lack of AI Certification Rules
No regulatory body certified the chatbot safe for mental health use.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
