This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Internal IBM documents revealed that the Watson for Oncology system provided unsafe and inaccurate cancer treatment recommendations, often deviating from standard medical guidelines. The system was trained on a limited set of synthetic, hypothetical patient cases rather than real-world clinical data, leading to physician dissatisfaction and concerns regarding the product's safety and reliability. Despite internal knowledge of these systemic flaws, IBM continued to market the product as being based on real patient data and consensus-driven expertise.
Internal documents from IBM Watson Health showed negative assessments from customers such as Florida’s Jupiter Hospital and Memorial Sloan Kettering criticizing its Watson for Oncology product for allegedly unsafe and incorrect cancer treatment recommendations.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
IBM was not transparent with users about the source of Watson's training data, falsely claiming it was based on real patient data and broad consensus.
The Watson for Oncology system failed to perform reliably, generating inaccurate and potentially dangerous treatment recommendations that deviated from standard medical guidelines.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The unsafe and inaccurate treatment recommendations were generated by the Watson for Oncology AI system after its deployment to hospitals worldwide.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
IBM Watson for Oncology provided unsafe and inaccurate cancer treatment recommendations due to being trained on synthetic, non-real-world patient data. While the incident caused significant reputational and financial damage to IBM and highlighted critical safety verification issues in medical AI, its direct national security impact is minor and restricted to technological competitiveness concerns.
The system acts as an Assistant, designed to provide treatment recommendations to supplement the clinical judgment of physicians. As stated in the report, 'treatment decisions ultimately require the involvement and clinical judgement of the treating physician' and 'the tool is intended to supplement – not replace — the clinical judgement of the treating physician.'
High Risk: The system is an AI tool used in healthcare for diagnostic and treatment recommendations. According to the definitions, High Risk includes 'AI used in healthcare, such as diagnostic tools.'
The Watson for Oncology system was designed to analyze patient data and recommend specific cancer treatment plans to assist oncologists, which directly aligns with Medical Diagnosis Support and acting as a Health AI Assistant.
The reports provide detailed, consistent information from internal IBM documents, customer feedback, and expert opinions regarding the system's training flaws and incorrect recommendations. There is clear consensus on the technical issues (synthetic data, small sample size) and the lack of reported physical harm to actual patients.
None. The failure of the AI system to provide safe recommendations due to poor training data is unambiguous.
Technology
Slow adaptation to new research
The software was slow to adapt to changing medical guidelines and research.
Inaccurate Recommendation Algorithm
The algorithm generated unsafe treatment options contrary to drug warnings.
Data Inputs
Use of Synthetic Training Data
Watson was trained on hypothetical cases instead of real-world patient data.
Small and Biased Dataset
Training used a small number of cases chosen without statistical input.
Narrow Expert Input
Training relied on the preferences of only one or two doctors per cancer.
Human Factors
Overreliance on Limited Expertise
Engineers relied on a few specialists instead of broad clinical guidelines.
Unconventional Clinical Judgement
Training doctors injected preferences that deviated from standard guidelines.
Process and Methods
Flawed Concordance Study Design
Validation studies were designed to make negative findings highly unlikely.
Lack of Statistical Validation
The training case sizes were determined without rigorous statistical input.
Management
Prioritization of Marketing
Executives promoted the tool for marketing despite knowing of systemic flaws.
Misleading Public Claims
Management falsely claimed the system was trained on real patient data.
Suppression of Internal Warnings
Executives ignored internal reports detailing unsafe medical recommendations.
Regulatory Environment
Lack of Strict Medical AI Oversight
Absence of regulatory checks allowed unsafe recommendations to be marketed.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
