This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The reports describe several downgraded AI safety incidents, notably an experiment where an OpenAI GPT-3 medical chatbot suggested a mock patient commit suicide, alongside an academic paper on the National Resident Matching Program algorithm.
Alvin Roth, a Ph.D at the University of Pittsburgh, describes the National Resident Matching Program (NRMP) and suggests future changes that are needed in the algorithm used to match recently graduated medical students to their residency programs.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The chatbot generated extremely harmful and unsafe content by suggesting that a mock patient should commit suicide.
The reports also detail a Tesla Autopilot lane recognition vulnerability to adversarial attacks.
The GPT-3 model demonstrated a critical lack of capability, consistency, and medical expertise during healthcare testing, leading to highly unsafe recommendations.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Pre-deployment
Occurring before the AI is deployed
The risk was caused by the GPT-3 model's output during an experimental pre-deployment evaluation by Nabla, where it unintentionally generated a harmful suggestion.
A collection of downgraded AI safety cases, including pre-deployment testing of a GPT-3 medical chatbot and academic research on computer vision vulnerabilities. Because these occurred in experimental, academic, or controlled settings with no real-world deployment or malicious intent, they carry negligible national security implications.
The report describes Nabla testing GPT-3's capabilities for 'medical documentation, diagnosis support, and treatment recommendation,' which are assistant functions designed to support healthcare professionals rather than act autonomously.
High Risk: The system was evaluated for 'diagnosis support, and treatment recommendation' in healthcare, which is a high-risk application under the EU AI Act due to its potential impact on patient safety.
The report explicitly states that Nabla researched GPT-3's capabilities for 'medical documentation, diagnosis support, and treatment recommendation' and created a 'medical chatbot' for the experiment.
The reports clearly state that the incidents described were downgraded because they represent experimental findings, academic studies, or projected risks rather than real-world harm events. This explicit context allows for a highly confident assessment that no actual harm occurred to real people.
One could interpret the reports as describing potential future harms, but the text explicitly confirms these are not real-world harm events.
Technology
Inconsistency in LLM Outputs
GPT-3 showed high inconsistency and lacked scientific and medical expertise.
Adversarial Input Vulnerability
Tesla Autopilot lane recognition was vulnerable to three small stickers.
Weakness in Semantic Reasoning
AI systems in the Winograd Challenge succeeded only 3% above random chance.
Data Inputs
Adversarial Physical Samples
Small stickers placed in intersections manipulated the lane recognition system.
Lack of Domain-Specific Data
GPT-3 lacked specialized medical training data needed for clinical tasks.
Manipulated Prompt Inputs
Human editors manipulated prompt inputs to generate sensationalist threats.
Human Factors
Active Human Manipulation
Human editors manipulated GPT-3 to exaggerate harm and generate threats.
Risky Experimental Testing
Mock patients or researchers tested the bot with dangerous scenarios.
Student Ethics Oversight Failure
A student deployed a facial recognition app without addressing privacy.
Process and Methods
Lack of Deployment Guardrails
Medical chatbot was tested without safety filters preventing harmful advice.
Inadequate Vulnerability Testing
Autopilot systems lacked robust testing against physical adversarial attacks.
Uncontrolled Campus Deployment
App was deployed on-campus before resolving ethical and privacy concerns.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
No population impact data reported.
