This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The South African Department of Home Affairs and the Department of Communications and Digital Technologies discovered that staff had included fictitious, AI-generated references in official policy documents. The incidents led to the suspension of senior officials and the initiation of independent reviews of all policy documents produced since late 2022. The departments maintain that the substantive policy positions remain accurate despite the erroneous citations.
South Africa's Department of Home Affairs reportedly appended apparent AI-generated fictitious references to its Cabinet-approved Revised White Paper on Citizenship, Immigration and Refugee Protection. The department said the references were not cited in the body, withdrew the standalone list, suspended two senior officials, and commissioned an independent review of policy documents produced since November 2022. It maintained that the paper's substantive policy positions were unaffected.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
Government officials relied on ChatGPT to generate factual references for official policy documents without verifying the outputs.
The AI system generated fictitious citations and references (hallucinations) which were included in official government policy documents, spreading false information.
Human
Due to a decision or action made by humans
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by government officials unintentionally publishing unverified, hallucinated AI outputs in official government policy documents after the AI system was publicly deployed.
Added categories peak at 2: Minor (Epistemic Harm, Direct)
South African government officials unintentionally included fictitious, AI-generated citations in official draft policy documents, including a white paper on immigration and the draft National AI Policy. This led to the suspension of officials and policy withdrawals, highlighting the risks of unverified AI use in state decision-making and policy formulation.
The report describes the AI system being used to draft policy documents and generate reference lists, acting as an assistant to augment human capabilities rather than taking independent action. The system required human oversight and decision-making, which failed in this instance.
Limited Risk: The system involved is ChatGPT, a general-purpose chatbot and text generator. According to the definitions, chatbots and systems generating AI content fall under Risk Level 3, requiring transparency obligations.
The AI system (ChatGPT) was used by government officials to assist in drafting policy papers and generating reference lists, which aligns with the functions of a Writing Assistant and Technical Text Generation.
The reports provide clear, consistent, and explicit details about the AI system used (ChatGPT), the nature of the failure (hallucinated references), the consequences (suspensions, policy withdrawals, independent reviews), and the affected government departments. There are no major conflicting accounts or significant ambiguities regarding the core events.
None
Technology
LLM Hallucinations
Large language models generated fictitious and erroneous citations.
Lack of Automated Verification Tools
No automated tools were used to verify the validity of the references.
Data Inputs
Unverified Reference Lists
Citations were generated and attached after the fact without verification.
Lack of Credible Data Sources
The LLM lacked access to or did not use credible data to generate sources.
Human Factors
Overreliance on AI Output
Officials trusted AI-generated citations without performing manual checks.
Lack of AI Literacy
Staff lacked understanding of LLM limitations like hallucination risks.
Process and Methods
Absence of AI Review Processes
Internal approval processes lacked specific AI checks and declarations.
Flawed Reference Compilation
References were attached after drafting without cross-referencing the text.
Management
Inadequate Oversight of Drafting
Management failed to supervise the drafting and referencing process.
Regulatory Environment
Lack of Government AI Guidelines
No clear public sector policies governed the safe use of AI in drafting.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
