This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Researchers tested two AI-powered interview platforms, MyInterview and Curious Thing, by providing responses in German rather than English. Both systems generated invalid personality assessments and English proficiency scores, demonstrating a lack of reliability and accuracy in their evaluation processes. The findings highlight concerns regarding the transparency and fairness of automated hiring tools.
Two AI interview softwares provided positive but invalid results such as "competent" English proficiency and high match percentage for interview responses given in German by reporters.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The systems unfairly represented candidates' traits based on irrelevant features like intonation, video backgrounds, and glasses, which can lead to biased hiring outcomes.
Employers are deploying and relying on these unvalidated AI tools for high-stakes hiring decisions despite their lack of scientific validity.
The AI systems failed to perform reliably under varying conditions, such as processing German inputs or ignoring irrelevant visual features like glasses and backgrounds, leading to invalid assessments.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The failures in candidate evaluation and language processing were caused by the deployed AI systems producing unexpected and invalid outputs when presented with German inputs.
An investigation into commercial AI hiring platforms revealed significant reliability and bias issues, such as failing to detect non-English inputs and altering scores based on irrelevant visual cues. While highlighting critical gaps in AI robustness and potential civil rights concerns regarding employment discrimination, the incident poses negligible direct threats to national security.
The AI systems function as assistants to human hiring managers, providing candidate scores and personality assessments to help them sift through applications, rather than making final hiring decisions autonomously.
High Risk: The report describes AI systems used in recruitment and employment evaluation ('AI-powered interview software claims to help employers sift through applications'), which falls under 'Employment and worker management systems, such as AI used in recruitment' in the High Risk category.
The primary goal of MyInterview and Curious Thing is to evaluate job applications and assess candidates' personality traits and skills during automated interviews.
The report provides clear, first-hand experimental evidence of the AI systems' failures when tested with German inputs, along with expert commentary and responses from the AI developers.
None
Technology
Unreliable Speech Recognition
System phonetically transcribed German words as nonsensical English words.
Invalid Intonation Analysis
Algorithm pulled personality traits from voice intonation, which is invalid.
Visual Background Bias
Algorithm altered candidate scores based on video backgrounds and glasses.
Data Inputs
Untested Language Inputs
Systems were not trained or tested on non-English inputs like German.
Unstructured Open-Ended Data
Open-ended audio responses provide unstructured data hard for AI to evaluate.
Human Factors
Overreliance on AI Scores
Hiring managers trust automated scores without verifying their validity.
User Misunderstanding of AI
Users assume AI evaluates actual job skills rather than superficial cues.
Process and Methods
Lack of Independent Testing
AI interviewing tools are not independently tested for accuracy or bias.
Flawed Validation Methods
Companies use unstandardized methods to measure complex traits.
Management
Prioritizing Efficiency over Accuracy
Firms adopt AI to sift applications quickly despite known reliability issues.
Lack of Algorithmic Transparency
Management keeps AI inner workings proprietary, preventing external scrutiny.
Regulatory Environment
Lack of AI Hiring Regulations
Absence of legal standards governing AI use in employment decisions.
No Oversight on Algorithmic Bias
Regulatory bodies do not audit automated hiring platforms for discrimination.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
