This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Researchers at Cornell University discovered that OpenAI's Whisper speech-to-text system frequently hallucinates violent language, fake websites, and fabricated personal information when processing audio containing long pauses. These hallucinations disproportionately affect individuals with speech impairments, such as those with aphasia. The study highlights significant risks for downstream applications in sensitive fields like legal, medical, and hiring processes.
Researchers at Cornell reportedly found that OpenAI's Whisper, a speech-to-text system, can hallucinate violent language and fabricated details, especially with long pauses in speech, such as from those with speech impairments. Analyzing 13,000 clips, they determined 1% contained harmful hallucinations. These errors pose risks in hiring, legal trials, and medical documentation. The study suggests improving model training to reduce these hallucinations for diverse speaking patterns.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
The system hallucinates violent language, potentially exposing users or downstream readers to harmful and inappropriate content.
The system fails to perform robustly under varying conditions, specifically when encountering silences or pauses in audio, leading to hallucinations.
Whisper performs unequally across groups because it is significantly more likely to hallucinate and produce errors when transcribing speech from individuals with speech impairments due to their longer pauses.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk is caused by Whisper's algorithmic hallucinations, which are an unexpected and unintentional outcome of the system's deployment.
Added categories peak at 2: Minor (Epistemic Harm, Inferred)
Cornell researchers found that OpenAI Whisper speech-to-text system frequently hallucinates violent language and fake information when processing audio with long pauses, disproportionately affecting speech-impaired individuals. While presenting safety and discrimination risks for downstream legal or medical use, the direct national security impact remains negligible.
Whisper operates as a Tool because it performs a specific, well-defined function of transcribing audio to text based on direct input without independent goal-directed planning or autonomous decision-making.
High Risk: The report describes Whisper's potential downstream use in high-risk areas such as 'AI-based hiring, in courtroom trials or patient notes in medical settings', which fall under the High Risk (Level 2) classification of the EU AI Act due to their significant implications for employment, justice, and healthcare.
The primary function of OpenAI's Whisper is speech-to-text transcription, which falls under Voice Recognition.
The report is based on peer-reviewed research presented at a major conference (FAccT) by Cornell University researchers. It provides specific statistics, such as a 1% hallucination rate across 13,000 tested clips, and clearly identifies the mechanism (long pauses triggering hallucinations). There is high certainty about the technical behavior and the affected demographic group.
None. The technical failure and its disproportionate impact on speech-impaired individuals are clearly documented.
Technology
Silence treated as words
The underlying LLM technology interprets silence or pauses as words.
Sensitivity to speech pauses
Whisper is more likely to hallucinate when speech has long pauses.
Data Inputs
Lack of diverse speech training
Training data did not adequately represent speakers with speech impairments.
Human Factors
User speech impairments
Aphasia and other conditions cause longer pauses, triggering hallucinations.
Process and Methods
Flawed modeling choices
Modeling choices failed to account for nonvocal durations during speech.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
