This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A government-deployed automated system used by the UK's Department for Work and Pensions (DWP) to flag housing benefit claims for fraud investigation significantly underperformed, incorrectly identifying over 200,000 legitimate claims as high risk. This resulted in unnecessary investigations, significant stress for claimants, and the waste of approximately £4.4 million in public funds. The incident has raised concerns regarding algorithmic bias, lack of transparency, and the potential for discriminatory impacts on vulnerable populations.
A Department for Work and Pensions (DWP) algorithm wrongly flagged over 200,000 UK housing benefit claims as high risk, resulting in unnecessary investigations. Two-thirds of these flagged claims were legitimate, causing wasted public funds and stress for claimants. Despite initial success in a pilot, the algorithm's real-world performance fell short. This incident highlights the risks of overreliance on automated systems in welfare administration.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.
Also classified as
The algorithm's predictive accuracy was nearly half of what was expected, resulting in a 66% error rate and 200,000 wrongful investigations.
The algorithm profiles individuals and determines fraud risk by weighing sensitive personal characteristics such as age and gender, leading to disproportionate and unfair targeting of vulnerable groups.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The high rate of false positives and subsequent wrongful investigations was an unexpected and unintended outcome of deploying the automated fraud detection system.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
The UK Department for Work and Pensions deployed an automated fraud-detection system that incorrectly flagged over 200,000 legitimate housing benefit claims. This resulted in 4.4 million pounds of wasted public funds on fruitless investigations and raised significant concerns about algorithmic bias and civil rights, representing a minor domestic governance and societal impact.
The system is described as an automated tool that performs a specific, deterministic function of flagging claims based on direct inputs without context retention or autonomous planning.
High Risk: The system is used by public authorities to evaluate eligibility for public assistance benefits, which is classified as high risk due to its significant impact on fundamental rights.
The system's primary function is to assign risk scores to housing benefit claimants to determine the likelihood of fraud or error, effectively scoring their trustworthiness.
The reports provide consistent, official figures obtained via Freedom of Information requests, clearly detailing the scale of the incident, the financial impact, and the demographic inputs used.
The incident could be interpreted as a standard bureaucratic or administrative failure rather than an AI safety incident, given the tool's lack of advanced machine learning capabilities.
Technology
Inaccurate Risk Prediction Algorithm
The algorithm was only 34-37 percent accurate compared to 64 percent in pilot.
Algorithmic Bias
Risk of bias against vulnerable groups with protected characteristics.
Invasive Profiling Tool
Mass profiling and citizen scoring based on personal characteristics.
Data Inputs
Flawed Profiling Criteria
Weighed personal data like age, gender, and number of children.
Discrepancy in Pilot Data
Pilot data showed 64 percent accuracy which failed to replicate in practice.
Human Factors
Overreliance on Automation
DWP prioritized automated flags leading to unnecessary investigations.
Lack of User-Centric Design
Failed to work with actual benefit claimants and users during development.
Process and Methods
Inadequate Validation Processes
Inefficient manual reviews of flagged cases costing 4.4 million pounds.
Lack of Transparency
Secretive algorithms used without public disclosure or clear explanations.
Management
Prioritizing Detection Over Rights
Focus on algorithm-led detection put vulnerable claimant rights last.
Premature System Expansion
Expanded AI deployment despite warnings of algorithmic bias.
Regulatory Environment
Lack of Independent Oversight
Inquiries failed to identify claimant harms despite high error rates.
Insufficient Legislative Safeguards
Attempts to push intrusive mass monitoring bills without enough protection.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
