This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The mental health nonprofit Koko faced significant public and expert criticism after it was revealed that they used OpenAI's GPT-3 to generate supportive messages for approximately 4,000 users without undergoing formal Institutional Review Board (IRB) ethical oversight. While the company claimed the feature was opt-in and disclosed, critics argued that experimenting on vulnerable individuals in a mental health context without rigorous informed consent processes poses serious ethical risks. The incident sparked a broader debate regarding the appropriateness of using AI for psychological support and the potential for harm when simulated empathy is introduced into sensitive human interactions.
OpenAI's GPT-3 was deployed by a mental health startup without ethical review to support peer-to-peer mental healthcare, and whose interactions with the help providers were "deceiving" for research participants.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
Also classified as
The AI system lacks the capability to feel or express genuine empathy, which led to users feeling that the interaction was 'weird' and 'empty' once they realized it was synthetic.
The incident involves the unsafe use of an AI chatbot in a highly sensitive mental health context, where the introduction of 'simulated empathy' was deemed inappropriate and ethically concerning for vulnerable users.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The controversy and ethical risks were caused by the intentional decision of Koko's founders to deploy GPT-3 for mental health support post-deployment.
Added categories peak at 2: Minor (Psychological Harm, Direct)
A mental health platform used GPT-3 to generate supportive messages for 4,000 users without formal ethical review board oversight. While raising significant ethical concerns regarding informed consent and human-AI interaction in sensitive contexts, the incident has negligible direct national security implications.
The AI was utilized in a 'co-pilot' fashion, meaning that a real human supervised the AI and could tweak its responses if necessary, augmenting human capabilities rather than acting independently.
Limited Risk: The system is a chatbot used to generate conversational text, which triggers specific transparency obligations under the EU AI Act to ensure users are informed they are interacting with AI.
The AI system (GPT-3) was used by peer supporters to assist them in writing and composing supportive messages for users experiencing mental health struggles.
The reports provide consistent details about the scale of the Koko experiment (4,000 users, 30,000 messages) and the specific AI model used (GPT-3). While there was initial confusion regarding whether users knew they were interacting with AI, the follow-up clarifications from the founder are well-documented, allowing for a reliable assessment of the ethical and operational boundaries crossed.
The incident could be interpreted as a benign pilot of a writing assistant tool that was mischaracterized on Twitter, rather than an unethical human-subject experiment.
Technology
Simulated Empathy
GPT-3 generated text that felt empty and weird to users once discovered.
LLM Limitations
LLMs generate plausible text without actual understanding or empathy.
Data Inputs
Lack of Stakeholder Data
Development lacked input from mental health experts and impacted communities.
Human Factors
User Backlash to AI
Users felt disturbed by machine-co-created messages in a sensitive context.
Process and Methods
No IRB Ethical Review
The experiment was not vetted by an Institutional Review Board before launch.
No Informed Consent
No formal consent process was provided to users experiencing crises.
Management
Efficiency over Ethics
Management prioritized response times over careful ethical review.
Premature Public Disclosure
Co-founder tweeted details of the AI trial, leading to public outrage.
Regulatory Environment
Regulatory Loophole
Private peer-to-peer apps are exempt from FDA IRB safety standards.
Unclear AI Regulations
Lack of clear legal frameworks for AI use in mental health support.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
