This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The Allegheny Family Screening Tool (AFST) is a predictive algorithm used by Allegheny County to prioritize child neglect reports for investigation. Independent analysis by the ACLU and ongoing federal investigations suggest the tool exhibits racial and disability-based bias, leading to disproportionate scrutiny of marginalized families. The tool's design decisions, such as using historical data from criminal and behavioral health systems, have been criticized for reinforcing systemic inequalities and lacking transparency for affected families.
Data analysis by the American Civil Liberty Union (ACLU) on Allegheny County's decision-support Family Screening Tool to predict child abuse or neglect risk found the tool resulting in higher screen-in rates for Black families and higher risk scores for households with disabled residents.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.
Also classified as
The tool's risk scores and underlying variables are opaque, leaving affected families unable to understand, question, or contest the decisions made about them.
The AFST algorithm systematically discriminates against Black and disabled families by assigning them higher risk scores, leading to unfair outcomes such as disproportionate child welfare investigations.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The discriminatory outcomes and biased risk scores were caused by the AFST AI system's predictions after it was deployed, though these risks were unintentional side effects of its design.
Added categories peak at 3: Substantial (Psychological Harm, Inferred)
A predictive child welfare algorithm in Pennsylvania exhibits systemic racial and disability bias, leading to civil rights concerns and federal investigations. While representing a serious local civil rights issue, the incident has negligible direct national security implications.
The AFST functions as an assistant because it generates risk scores to aid human call-screeners in making decisions, rather than autonomously executing child removals without human oversight.
High Risk: The system is used by public authorities in a high-stakes social services context (child welfare risk assessment) which has significant implications for fundamental rights and safety.
The primary goal of the AFST is to assess the risk of child neglect and predict the likelihood of child removal within two years using behavioral and administrative data.
The reports provide detailed, credible accounts from both the affected family (the Hackneys) and an independent audit by the ACLU. The technical design choices of the AFST (V2.1) and their discriminatory impacts are thoroughly documented, and the ongoing federal investigation provides additional external validation of the safety risks.
One could argue the child removal was entirely due to the hospital's referral and caseworker judgment rather than the AFST, though the tool's mandatory screening protocols for high scores suggest it played a significant role.
Technology
Misleading Risk Grouping
Tool displays only the highest score, treating families as risky by association.
Arbitrary Feature Selection
Thresholds for selecting variables were based on trial and error.
Algorithmic Opacity
The risk calculation details and scores are kept hidden from families.
Data Inputs
Biased Historical Records
Juvenile justice and jail databases overrepresent marginalized groups.
Disproportionate Health Data
Behavioral health data penalizes families seeking disability-related care.
Socioeconomic Data Skew
Lack of private healthcare data overrepresents low-income families.
Human Factors
Automation Bias
Screeners treat mathematical outputs as objective, unquestionable truths.
Lack of Recourse
Affected families cannot access or challenge their assigned risk scores.
Screener Information Deficit
Call screeners only see aggregated household labels rather than details.
Process and Methods
Mandatory Screen-in Protocols
High scores automatically trigger mandatory investigations with little override.
Unchangeable Past Factors
Using permanent historical records offers families no way to escape their past.
Inadequate System Audits
The tool was deployed without addressing potential equity and bias impacts.
Management
Ignoring Research Feedback
County and developers ignored independent critical audits of the tool.
Prioritizing Efficiency
Focusing on screening speed and cash-strapped budgets over equity.
Lack of System Transparency
Management refused to disclose individual scores or explain variables used.
Regulatory Environment
Lack of Algorithmic Standards
No federal standards exist to regulate AI tools in child welfare systems.
Post-Hoc Enforcement
Federal civil rights agencies only investigate after harm has occurred.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
