This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A child protection worker in Victoria, Australia, used ChatGPT to draft a court-submitted Protection Application Report, inputting sensitive case details. The AI-generated content contained significant inaccuracies, including mischaracterizing evidence of sexual abuse as positive parenting behavior. This incident led to an investigation by the Office of the Victorian Information Commissioner (OVIC), which found that the department failed to protect sensitive information and issued a compliance notice requiring a ban on the use of generative AI tools by child protection staff.
A child protection worker in Victoria, Australia reportedly used ChatGPT to draft a report submitted to the Children's Court. The purportedly AI-generated report contained inaccuracies and downplayed risks to the child, allegedly resulting in a privacy breach when sensitive information was shared with OpenAI.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
Also classified as
The worker disclosed highly sensitive personal information of a child, parents, and carer to OpenAI by inputting it into ChatGPT.
ChatGPT failed to understand the context of sexual abuse and hallucinated that a doll used for sexual purposes was an indication of positive parenting.
The worker inappropriately relied on a general-purpose chatbot to draft a highly sensitive child protection risk assessment in a critical legal context, leading to unsafe and inaccurate outputs.
Human
Due to a decision or action made by humans
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was initiated by a human worker's decision to use an unauthorized tool and input sensitive data, with the resulting privacy breach and inaccurate report being unintentional outcomes of trying to save time.
A child protection worker in Australia used ChatGPT to draft a sensitive court report, leaking private data and generating severe hallucinations that downplayed child abuse. The incident highlights established risks of shadow AI use in government processes, resulting in domestic regulatory action but presenting negligible direct threat to national security.
ChatGPT acted as an assistant to the child protection worker, who used it to draft the Protection Application Report to save time and present work more professionally, requiring human oversight which was initially lacking.
Limited Risk: ChatGPT is a chatbot, which falls under Risk Level 3 requiring specific transparency obligations, though its unsafe use in a child protection context highlights high-risk implications.
The worker used ChatGPT as a writing assistant to draft a Protection Application Report, which is a form of child welfare risk assessment.
The reports are highly detailed and originate from the official investigation by the Office of the Victorian Information Commissioner (OVIC), providing clear facts, specific quotes, and regulatory findings.
None. The facts of the unauthorized use and the specific hallucination are well-documented and accepted by the department.
Technology
Probabilistic Nature of LLMs
LLMs prioritize sequence probability over factual accuracy and context.
Inability to Understand Context
ChatGPT failed to grasp the context of sexual abuse, misinterpreting a risk.
Inaccurate Content Generation
The AI hallucinated positive parenting strengths from details of abuse.
Data Inputs
Input of Sensitive Client Data
Worker entered names and risk assessments into a public AI tool.
Data Shared with Third Party
Sensitive data was disclosed to OpenAI, releasing it from DFFH control.
Human Factors
Desire to Save Time
Worker used ChatGPT to save time and present work more professionally.
Lack of AI Literacy
Staff did not understand risks of entering sensitive data into public AI.
Over-reliance on AI Output
Worker trusted and submitted AI-generated text containing critical errors.
Process and Methods
Lack of Technical Access Controls
DFFH had no IP or DNS blocking to prevent access to public GenAI tools.
Inadequate Review of Reports
Draft with major AI errors was submitted to court before legal review.
No Dedicated GenAI Policies
DFFH had no specific rules regarding when and how GenAI should be used.
Management
Lack of Employee Training
Management failed to provide training on GenAI privacy and accuracy risks.
Insufficient Risk Assessment
Management did not assess risks of GenAI across different functions.
No Visibility into AI Usage
DFFH had no way to monitor what data employees entered into GenAI tools.
Regulatory Environment
Inadequate External AI Regulations
Lack of strict statutory rules governing public sector use of GenAI.
Privacy Principle Contravention
DFFH failed to meet IPP 3.1 and 4.1 standards for data protection.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
