This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Meta AI released a public demo of Galactica, a large language model for scientific literature, which was quickly criticized for generating authoritative-sounding misinformation and fabricated research. The model also exhibited biased outputs and implemented opaque safety filters that censored legitimate scientific topics. Following intense backlash from the scientific community, Meta took the demo offline three days after its launch.
Meta AI trained and hosted a scientific paper generator that sometimes produced bad science and prohibited queries on topics and groups that are likely to produce offensive or harmful content.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
The model's safety filters censored queries about AIDS and queer theory, which critics argued further marginalized and misrepresented minority communities.
The model exhibited a fundamental lack of capability to distinguish truth from falsehood, leading to hallucinations and incorrect scientific outputs.
The primary issue was Galactica generating highly authoritative-sounding but completely false scientific information, fake papers, and incorrect facts, which could mislead users.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The generation of false scientific facts and biased content was an unexpected outcome of the AI system's operation after it was deployed as a public demo.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
Meta AI's short-lived public release of the Galactica LLM highlighted risks of automated scientific misinformation and biased outputs. While the incident caused epistemic concern within the scientific community and minor societal concerns due to biased filtering, it posed negligible threats to core national security areas like critical infrastructure, sovereignty, or intelligence.
The model was designed to augment human capabilities by summarizing papers, writing code, and generating text, requiring human oversight to verify its outputs.
Limited Risk: The system is a generative AI chatbot. Under the EU AI Act, chatbots and generative models are classified as Limited Risk, requiring transparency to inform users they are interacting with AI.
The model was explicitly designed and promoted as a writing assistant for scientific literature, capable of summarizing papers, writing code, and generating text to help researchers.
The reports provide consistent, detailed accounts of the model's release, its specific failures (hallucinations, censorship, bias), and its subsequent takedown. There is high agreement among the sources.
The incident could be interpreted primarily as a lack of capability (7.3) rather than misinformation (3.1), but the core risk realized was the generation of false information.
Technology
Stochastic Parrot Limitations
Model captures word patterns but cannot tell fact from fiction.
Flawed Content Filtering
Filters censored benign minority topics while letting toxic prompts pass.
Lack of Output Provenance
The UI lacked features to flag or identify model-generated content.
Data Inputs
Statistical Pattern Training
Training on 48M papers led to mimicry rather than semantic understanding.
Inherent Training Data Biases
The training data led to the reproduction of prejudice and stereotypes.
Human Factors
Adversarial Prompting by Users
Users entered racist or offensive prompts to test model limits.
Blind Trust in Model Outputs
Users may treat outputs as authoritative without verifying facts.
Defensive Developer Attitude
Management blamed users for 'casually misusing' the demo.
Process and Methods
Misleading Product Hype
Marketing framed a research model as a solved scientific tool.
Inadequate UI Safeguards
The UI styled outputs like real papers without clear disclaimers.
Poor Issue Reporting Channels
No structured way for community feedback other than public Twitter.
Management
Hasty Public Deployment
Released a public demo without thoroughly assessing safety risks.
Unclear Use Case Definition
Failed to clearly document and restrict intended use cases.
Regulatory Environment
Lack of Generative AI Standards
No clear regulatory framework governs the release of LLM demos.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
