This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
An unregulated AI medical transcription scribe hallucinated a false claim that a patient micro-dosed psychedelic mushrooms, which was then included in her official medical record without the doctor's verification.
Australian patient Rebecca Green reportedly discovered that a post-operative letter sent by her urologist to her GP falsely said she micro-dosed psychedelic mushrooms and suggested this could explain prior kidney bleeding. Green had reportedly consented to an AI scribe during her first urology appointment and feared the false record could affect her workers' compensation case. Her urologist reportedly apologized, corrected the letter, and said the error appeared to arise during dictation or transcription.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The AI system generated discrete false and misleading information regarding the patient's drug habits, which polluted her official medical record.
The doctor overrelied on the AI scribe by failing to check the output before sending it, illustrating the human tendency to stop verifying AI outputs over time.
The primary failure lies within the AI system's lack of capability and robustness, specifically its propensity to hallucinate false symptoms, conditions, and drug use during transcription.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by an AI scribe hallucinating false medical information during post-deployment operation, which was an unexpected and unintentional outcome of its transcription goal.
Added categories peak at 4: Severe (Epistemic Harm, Inferred)
An unregulated AI medical transcription tool hallucinated false drug use in an Australian patient's record due to a doctor's failure to verify the output. While highlighting significant clinical safety, regulatory, and privacy concerns in healthcare AI adoption, the incident poses negligible threat to national security.
The AI scribe acts as an Assistant because it is designed to augment the doctor's capabilities by transcribing and summarizing appointments, but it requires human oversight and validation before the output is finalized.
High Risk: The AI system is used in healthcare as a clinical documentation tool, which has significant implications for patient safety and falls under the category of AI used in healthcare.
The primary function of the AI scribe is to transcribe and summarize spoken medical appointments, acting as a writing and dictation assistant for clinicians.
The report provides clear, firsthand accounts of the patient's experience, quotes from medical professionals, and references to a watchdog report by Digital Rights Watch, making the details highly reliable.
None
Technology
AI Hallucination
The AI scribe made up a false claim that the patient took illegal drugs.
Algorithmic Inaccuracy
The AI tool confused body sides and clinical details like breast cancer location.
Unreliable Transcription
The transcription process introduced errors that sounded plausible but were false.
Data Inputs
Acoustic Transcription Errors
The tool misheard spoken words during the doctor-patient consultation.
Lack of Input Verification
The software did not verify transcribed facts against clinical evidence.
Human Factors
Automation Bias
The doctor stopped checking the AI output after initial accurate experiences.
Coerced Patient Consent
The patient agreed to AI use to avoid being perceived as a difficult patient.
Inadequate Physician Review
The urologist failed to review the post-operative letter before sending it.
Process and Methods
Lack of Output Verification Process
There was no mandatory verification process for AI outputs before clinical use.
Inconsistent Consent Methods
Consent was obtained through passive signs or forced online booking systems.
Deficient Error Correction Flow
Patients faced significant frustration finding and correcting medical errors.
Management
Inadequate Practice Governance
The medical practice lacked policies ensuring clinicians checked AI outputs.
Prioritizing Speed Over Safety
The practice adopted AI to reduce admin work without maintaining safety checks.
Regulatory Environment
Exemption from Oversight
Scribes claiming only to transcribe are exempt from TGA medical device rules.
Lack of Approved Software Lists
Australia lacked a centralized list of tested and approved AI medical tools.
Unregulated Medical Scribes
The AI scribes market operated without mandatory safety testing standards.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
