This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A class-action lawsuit alleges that State Farm's automated claims processing system, which utilizes AI software from FRISS, discriminates against Black policyholders. The system allegedly uses proxies for race, such as neighborhood urbanization data, to flag claims for potential fraud, leading to disproportionate delays and increased administrative requirements for Black customers compared to white customers.
State Farm's automated claims processing method was alleged in a class action lawsuit to have disproportionately against Black policyholders when paying out insurance claims.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
The AI system uses neighborhood and demographic proxies that result in systemic racial discrimination against Black homeowners.
The proprietary nature of the FRISS algorithm and State Farm's claims data makes it difficult for policyholders to understand or challenge their risk scores.
The automated fraud detection system performs with unequal effectiveness across demographic groups, flagging Black policyholders at a significantly higher rate and causing them unequal outcomes and increased administrative burdens.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The discriminatory outcomes were caused by the post-deployment operation of the FRISS AI fraud detection system, which unintentionally generated racially biased risk scores.
Added categories peak at 2: Minor (Psychological Harm, Indirect)
A federal class-action lawsuit alleges racial discrimination in State Farm's automated claims processing system, which uses FRISS AI software. The system allegedly uses geographic and demographic proxies to disproportionately flag Black policyholders' claims for fraud. This incident highlights domestic algorithmic bias and civil rights concerns but carries negligible direct implications for national security.
The FRISS AI system acts as an Assistant by analyzing policyholder data to generate a risk score and flag potential fraud, which is then used by human investigators to make final decisions.
Risk Level 2 (High Risk): The AI system is used for risk evaluation and fraud detection in housing insurance, which has significant implications for fundamental rights (non-discrimination) and access to essential financial services.
The FRISS AI system is designed to analyze customer profiles and claim narratives to generate a risk score and flag potential fraud, which is a form of trustworthiness scoring.
The reports are detailed, citing a specific federal class-action lawsuit, a structured statistical study of 800 participants, and specific details of the AI software (FRISS) and its inputs (urbanization, social media data).
State Farm could argue the delays are due to human investigator bias in the special unit rather than the algorithmic scoring itself, though the lawsuit targets both.
Technology
Proxy Variable Correlation
Algorithms learn to use zip codes and urbanization as proxies for race.
Biased FRISS Risk Scoring
AI software flags claims for potential fraud using demographic risk scores.
Data Inputs
Biased Demographic Inputs
Using urbanization and crime statistics which correlate heavily with race.
Social Media Harvesting
Harvesting social media data which may contain inherent racial biases.
Human Factors
Investigator Racial Bias
Special unit managers single out Black neighborhoods for fraud investigation.
Over-reliance on AI Flags
Adjusters rely on automated scores, leading to increased administrative burden.
Process and Methods
Automated Fraud Flagging
Routing claims to special units based on opaque, automated risk scores.
Lack of Algorithmic Auditing
State Farm failed to proactively audit automated systems for racial bias.
Management
Failure to Monitor AI Bias
Management did not examine algorithmic bias despite knowing the risks.
Prioritizing Cost Reduction
Deploying automated claims processing to cut labor costs without safeguards.
Regulatory Environment
Weak AI Regulatory Oversight
Lack of clear regulations governing AI use and bias in insurance claims.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
