This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
During the COVID-19 pandemic, hundreds of AI-based diagnostic and triage tools were developed and deployed in clinical settings. Multiple large-scale studies, including reviews by the Turing Institute and researchers at Maastricht University and the University of Cambridge, concluded that none of these tools were fit for clinical use due to poor data quality, flawed training methodologies, and lack of clinical validation. Experts expressed significant concern that these tools, some of which were marketed to hospitals, may have caused patient harm by providing inaccurate diagnostic information.
AI tools failed to sufficiently predict COVID patients, some potentially harmful.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
Hospitals and clinicians deployed and relied on these unvalidated AI tools in critical clinical situations before they were ready.
The AI diagnostic tools lacked the capability and robustness required for clinical use, failing to perform reliably due to flawed training methodologies and poor data quality.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk of patient misdiagnosis was caused by the AI systems' incorrect outputs, which was an unintentional outcome of trying to help clinical triage, occurring after these models were deployed or marketed to hospitals.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
During the COVID-19 pandemic, hundreds of unvalidated AI diagnostic and triage tools were deployed in clinical settings globally. Flawed training methodologies and data quality issues rendered them unfit for clinical use, posing minor national security risks by potentially compromising critical healthcare infrastructure and patient safety during a public health crisis.
The AI models were developed to assist clinicians in diagnosing COVID-19 and predicting patient risk from medical images, acting as clinical assistants to support human decision-making.
High Risk: The report describes deep-learning models for diagnosing covid and predicting patient risk from medical images, which falls under AI used in healthcare, such as diagnostic tools.
The primary goal of the developed AI systems was to diagnose COVID-19 and predict patient risk to support clinical decision-making.
The reports provide high-quality, peer-reviewed evidence from major studies (BMJ, Nature Machine Intelligence) detailing the exact failure modes of the AI models. While the exact number of patients affected is not quantified, the technical and systemic aspects of the failure are exceptionally well-documented.
None
Technology
Flawed Model Training
Models learned patient position or text fonts instead of actual clinical signs.
Overfitting on Duplicated Data
Frankenstein datasets caused models to be tested on their own training data.
Data Inputs
Poor Quality Frankenstein Datasets
Datasets spliced from multiple sources contained duplicates and unknown origins.
Biased Training Images
Healthy control scans were children, teaching AI to detect age, not COVID.
Incorporation Bias in Labels
Scans were labeled by doctor opinion rather than objective PCR tests.
Human Factors
Lack of Cross-Disciplinary Skills
AI developers lacked clinical insight and medical experts lacked math skills.
Hype and Unrealistic Expectations
High hype encouraged deployment of unready tools, risking patient safety.
Process and Methods
Lack of Model Validation
Researchers rushed to build new models instead of validating existing ones.
Non-Standardized Data Formats
Absence of standardized data formats hindered effective data sharing.
Management
Secrecy and NDAs
Hospitals signed NDAs with vendors, preventing independent model evaluation.
Poor Academic Incentives
Lack of career incentives to share work or validate existing clinical models.
Regulatory Environment
Lack of Clinical Oversight
Tools were marketed and used without rigorous clinical validation or standards.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
