This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The Oregon Department of Human Services discontinued its 'Safety at Screening Tool' algorithm following concerns regarding racial bias and its impact on child welfare investigations. The tool, which was designed to predict child risk, was replaced by a new process to improve equitable decision-making. This decision followed broader scrutiny of similar algorithmic tools in other states, which were found to disproportionately flag Black families for investigation.
Oregon’s Department of Human Services (DHS) stopped using its Safety at Screening Tool, that is aimed to predict the risk that children wind up in foster care or be investigated in the future, and opted for a new process allegedly to reduce disparities and improve racially equitable decision-making.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
The systemic over-flagging of Black families led to unfair outcomes and unequal treatment by child protective services.
The algorithm exhibited unequal performance and predictive disparities across demographic groups, disproportionately flagging Black families for child neglect investigations.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The racial disparities and bias in the child welfare screening decisions were caused by the deployed AI algorithm, which was an unintentional and unexpected outcome of its deployment.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
The Oregon Department of Human Services discontinued its child welfare screening algorithm due to concerns over racial bias. While the incident highlights significant civil rights and societal concerns regarding algorithmic fairness in public administration, its national security impact remains negligible to minor, with no threats to physical security, sovereignty, or critical infrastructure.
The AI system acts as an Assistant because it generates numerical risk scores to help social workers decide whether to investigate a family, augmenting human decision-making rather than taking autonomous action. The report notes that 'social workers can always override the tool, which was never intended to be used on its own'.
High Risk: The system is used by public authorities in child welfare to make critical decisions about family intervention and child safety, which has significant implications for fundamental rights and public interests. The report notes that 'making decisions about what should happen to children and families is far too important a task to give untested algorithms'.
The primary goal of the Safety at Screening Tool was to assess the risk of child abuse or neglect to help social workers decide which families to investigate, which is a child welfare risk assessment function.
The report clearly outlines the decision to discontinue the algorithm, the reasons (racial bias and disparities), and the context of similar tools in other states. While specific statistical data on the exact number of affected families is missing, the overall narrative and policy implications are well-documented.
None. The discontinuation of the tool due to racial bias concerns is clear and uncontested.
Technology
Algorithmic Racial Bias
The algorithm flagged a disproportionate number of Black families.
Untested Predictive Models
High-stakes decisions were given to untested screening algorithms.
Reliability Concerns
There were concerns about the reliability of the predictive technology.
Data Inputs
Inadequate Data
The reunification pilot was paused due to inadequate data.
Biased Internal Data
The model drew from internal child welfare data which may reflect bias.
Human Factors
Hardening of Human Bias
Social workers viewing scores could have their existing biases hardened.
Process and Methods
Lack of System Transparency
There was a lack of transparency in how the algorithm made decisions.
Flawed Screening Process
The screening process did not ensure racially equitable decisions.
Management
Adoption of Flawed Models
Management adopted a model inspired by another state's biased tool.
High Leadership Turnover
Agency prepared to hire its eighth director in six years, limiting oversight.
Regulatory Environment
Lack of National Oversight
Absence of federal regulations and oversight for child welfare algorithms.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
