This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The 'Robodebt' scandal involved an automated compliance system that incorrectly issued debt notices to hundreds of thousands of Australians. By using flawed income-averaging algorithms to compare tax and welfare data, the system falsely accused individuals of welfare fraud. The program caused widespread financial and psychological harm, leading to a $1.8 billion class-action settlement and a Royal Commission.
Australian Department of Human Services (DHS)’s automated debt assessment system issued false or incorrect debt notices to hundreds of thousands of people, resulting in years-long lawsuits and damages to welfare recipients.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The government removed human review from the debt-raising process, delegating the authority to issue debt notices entirely to the automated system without meaningful human oversight.
The program represents a massive failure of public administration and regulatory oversight, where warnings about the system's illegality were ignored for years.
The core failure was the algorithmic system's inability to accurately calculate fortnightly income from annual data, making it fundamentally non-robust and prone to massive errors when applied to casual workers.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The financial and psychological harms resulted from the automated decisions of the Robodebt algorithm post-deployment, which produced unexpected and incorrect debt calculations while pursuing the goal of welfare compliance.
Added categories peak at 4: Severe (Psychological Harm, Direct)
The Robodebt scandal represents a major domestic governance and human rights failure driven by a fully autonomous government algorithm. By unlawfully targeting 443,000 citizens with flawed debt notices, the system caused severe psychological and financial harm, leading to a Royal Commission and a 1.8 billion dollar settlement. It highlights the profound risks of removing human oversight from automated state decision-making.
The system operated independently to match data, generate letters, and raise debts without human intervention. As reported, 'the computer prints out and sends the letter on its own' and 'remove humans entirely from the process of identifying alleged debts'.
High Risk: The system is an AI application used by public authorities to evaluate eligibility for public assistance benefits and to reclaim such benefits, which has significant implications for fundamental rights and public interests.
The primary goal of the system was to process financial data from the ATO and Centrelink to identify and recover welfare overpayments, thereby managing public resource allocation.
The reports provide extensive, consistent, and highly detailed coverage of the Robodebt scandal, including court rulings, government admissions, and specific case studies. The legal and operational facts of the system's failure are well-documented.
None. The system's failure and unlawfulness are established by federal court consent orders and government admissions.
Technology
Flawed Income Averaging Algorithm
Divided annual ATO income into 26 fortnights, misrepresenting casual work.
Automated Letter Dispatch
System automatically generated and sent debt notices without human review.
Employer Double-Counting
Failed to match spelling variants of employer names, doubling reported income.
Data Inputs
Mismatched Data Granularity
Compared annual tax data with fortnightly welfare reporting intervals.
Outdated Customer Addresses
Sent letters to old addresses, causing default debts without user knowledge.
Lack of Actual Income Proof
Relied on averaged tax data rather than verifying actual fortnightly earnings.
Human Factors
Cognitive Bias in System Output
Staff and ministers assumed algorithmic outputs were correct without checking.
Shifting Burden of Proof
Required vulnerable citizens to retrieve payslips from up to six years ago.
Staff Quotas and Pressure
Staff pressured to process high volumes of debt notices with little training.
Process and Methods
Removal of Human Oversight
Replaced manual officer investigations with fully automated debt generation.
Inadequate Internal Reviews
Forced citizens to repay disputed debts while slow reviews were underway.
Inaccessible Support Channels
Blocked phone lines and online portal crashes prevented users from disputing.
Management
Prioritizing Budget Savings
Focused on clawing back billions over ensuring system accuracy and fairness.
Stonewalling and Denial
Defended the system publicly and ignored warnings from welfare and legal groups.
Inadequate Risk Assessment
Failed to assess legal and psychological risks before turbocharging automation.
Regulatory Environment
Unlawful Interpretation of Law
Social Security Act did not permit raising debts based on income averaging.
Ignoring Tribunal Rulings
Department ignored early Administrative Appeals Tribunal rulings of illegality.
Lack of External Audit
No independent algorithmic audits were conducted before scaling the program.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
