This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
An investigation by the AI Democracy Projects and Factchequeado tested five leading AI models on their ability to answer election-related questions in English and Spanish. The study found that models frequently provided inaccurate or misleading information, with a notable disparity in performance where Spanish-language queries were significantly more likely to result in errors or irrelevant information about foreign elections. The findings highlight ongoing concerns regarding the reliability of AI as a source of information for voters in the U.S.
An analysis reportedly found that multiple AI models provided inaccurate responses to election-related questions, with 52% of Spanish-language answers and 43% of English-language answers containing misinformation or omissions. Errors included misidentifying voting processes and providing information about foreign elections.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
The models frequently generated false, misleading, or incomplete information regarding U.S. election processes, candidates, and voting security.
The AI models exhibited a significant disparity in accuracy rates between English (43% inaccurate) and Spanish (52% inaccurate) queries, resulting in unequal performance for Spanish-speaking users seeking election information.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The inaccuracies and language disparities were produced by deployed AI models responding to user queries, representing an unintended failure mode of the systems.
Added categories peak at 4: Severe (Epistemic Harm, Inferred)
An empirical study revealed that five leading AI models frequently provide inaccurate election information, with a higher error rate for Spanish-language queries than English. This systemic failure poses a substantial threat to sovereignty and societal stability by potentially disenfranchising Spanish-speaking voters and undermining public trust in democratic processes.
The AI models function as conversational assistants, responding to user queries about elections by providing information and suggestions, which requires human verification and oversight to ensure accuracy.
Limited Risk: The systems evaluated are general-purpose chatbots, which are classified as limited risk under the EU AI Act, requiring transparency so users know they are interacting with AI.
The AI systems evaluated are general-purpose chatbots designed to interact with users and answer questions on a wide range of topics, including elections.
The report is based on a structured, empirical study conducted by the AI Democracy Projects and Factchequeado, providing clear statistics (e.g., 52% vs 43% inaccuracy rates) and specific examples of model failures across multiple languages. The roles of the AI systems and the nature of the failures are explicitly documented.
The incident could be viewed primarily as a general capability limitation (7.3) rather than unequal performance (1.3), but the significant statistical disparity between English and Spanish performance strongly supports the discrimination/unequal performance classification.
Technology
Language Bias in Model Training
Models like Claude 3 were trained primarily on English language data.
Failure to Detect Spanish Queries
Systems failed to recognize election queries when asked in Spanish.
Algorithmic Hallucination of Links
Models generated broken links or English links in Spanish responses.
Data Inputs
Lack of Spanish Election Data
Lack of Spanish data led to referencing Latin American elections.
Vulnerability to Online Misinfo
Models absorbed unverified claims where truthful online data was scarce.
Incomplete Context in Training Data
Omission of crucial localized voting facts like AZ live video feed rules.
Human Factors
Developer Neglect of Non-English
Developers prioritized English performance, neglecting safety in Spanish.
User Trust in Chatbot Accuracy
Users rely on chatbots for critical voting info despite high error rates.
Process and Methods
Inadequate Multi-language Testing
Companies failed to test safety guardrails across multiple languages.
Failed Redirection Mechanisms
Pledges to redirect users to TurboVote or CanIVote were not executed.
Poor Quality Control on Outputs
Lack of validation allowed models to output contradictory safety claims.
Management
Premature Model Release
Releasing models for public use before verifying election-related safety.
Neglecting Developer Guidelines
Meta released Llama 3 without including election safety in developer guides.
Broken Safety Pledges
Management failed to enforce commitments to redirect election queries.
Regulatory Environment
Lack of Standardized AI Mandates
No regulatory mandates forced companies to ensure bilingual election safety.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
