This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
During a workplace training session at Bunbury Prison, a trainer used Microsoft's Copilot to generate case study scenarios. The AI produced a scenario that included the real name of a former employee involved in an active sexual harassment case, despite claiming the scenario was fictional. This incident highlights the risks of AI 'regurgitation' of sensitive, publicly available information when used for content generation.
During workplace training at Bunbury Prison in Western Australia, a trainer used Microsoft's Copilot AI chatbot to generate case study scenarios. The chatbot produced a scenario that included the real name of a former employee involved in a sexual harassment case, revealing sensitive information.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The AI system exposed users in a workplace training environment to sensitive, triggering, and inappropriate real-world sexual harassment details involving their former colleague.
The AI system explicitly generated false meta-information by claiming 'this case study is entirely fictional' when it actually contained real names and active court details, misleading the user.
The AI system failed to perform reliably by failing to adhere to the negative constraint of generating a fictional scenario, instead regurgitating real-world sensitive data and falsely labeling it as fictional.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by Microsoft's Copilot chatbot generating real-world names and details while claiming the scenario was fictional, which was an unexpected and unintentional outcome of the generation process.
Added categories peak at 2: Minor (Psychological Harm, Direct)
A Microsoft Copilot chatbot regurgitated the real name and details of a harassment victim during a prison staff training session in Australia. While causing localized distress and highlighting LLM data privacy risks, the incident poses negligible threats to national security.
The AI system, Microsoft Copilot, acted as an Assistant by generating case study scenarios based on user prompts to help the trainer prepare course materials. It did not take independent action or make autonomous decisions outside of responding to the user's request.
Risk Level 3. Limited Risk: The system involved is a chatbot (Microsoft Copilot) generating text content. According to the definitions, chatbots and AI-generated content systems pose a moderate risk and are subject to transparency obligations to ensure users are aware they are interacting with AI.
The system is Microsoft's Copilot, a general-purpose chatbot used here as a writing assistant to generate fictional case studies for a training course.
The report clearly details the AI system used (Microsoft Copilot), the developer (Microsoft), the context of the incident (workplace training at Bunbury prison), the specific output failure (regurgitating real names/details while claiming to be fictional), and the direct psychological impact on the victim. The ABC's independent replication provides strong corroboration of the system's behavior.
One could argue this is a privacy incident, but since the information was publicly available from court proceedings, it is classified under capability failure and epistemic/toxic content rather than a privacy breach per the boundary rules.
Technology
Regurgitation of Training Data
Chatbot outputted real names and case details instead of fictional ones.
Misleading System Assurances
AI falsely claimed the generated case study was 'entirely fictional'.
Lack of Output Verification
No built-in safety filter to prevent leaking real-world litigation names.
Data Inputs
Ingestion of Active Court Data
System trained on public search data containing sensitive court cases.
Specific Prompting Vulnerability
Detailed prompts caused chatbot to retrieve and insert real people's names.
Human Factors
Overreliance on AI Output
Trainer assumed the generated scenario was fictional without verifying.
Lack of AI Literacy
Trainer did not know generative AI could hallucinate or regurgitate data.
Process and Methods
Inadequate Content Review
No validation of training materials before presenting them to employees.
Lack of AI Guidelines
Department lacked policies on prompting and checking AI outputs.
Management
Inadequate Vendor Oversight
Department failed to audit training materials provided by the contractor.
Lack of AI Policy
Organization did not establish rules for using generative AI in training.
Regulatory Environment
Absence of AI Safety Standards
No clear regulatory frameworks preventing regurgitation of sensitive data.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
