This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Peer reviewers for the Australian Research Council (ARC) allegedly used ChatGPT to generate feedback on confidential government grant applications. This practice led to the unauthorized uploading of sensitive research proposals into an external AI system, raising significant concerns regarding confidentiality breaches and academic misconduct. The ARC subsequently issued a warning that such use of AI violates existing confidentiality policies.
Peer reviewers of Australian government grant applications inserted applicants' work into generative AI systems such as ChatGPT to generate assessment reports, which allegedly posed confidentiality and security issues.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that memorize and leak sensitive personal data or infer private information about individuals without their consent. Unexpected or unauthorized sharing of data and information can compromise user expectation of privacy, assist identity theft, or cause loss of confidential intellectual property.
Also classified as
Reviewers inappropriately relied on ChatGPT to generate academic assessments, substituting automated text generation for rigorous human peer review.
The incident involved the unauthorized uploading of confidential, unpublished grant proposals containing sensitive research ideas to an external AI system, violating confidentiality agreements.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The confidentiality breach and poor-quality reviews were caused by human peer reviewers intentionally choosing to upload sensitive proposals to ChatGPT post-deployment.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
Australian Research Council peer reviewers compromised the confidentiality of sensitive, unpublished grant proposals by uploading them to ChatGPT. While this represents a minor breach of government information security and compromises the integrity of the public funding process, the national security impact remains low and manageable through updated policy enforcement.
The peer reviewers used ChatGPT as a writing assistant to generate draft text and summaries of the grant proposals, which they then submitted as their own peer review reports.
Risk Level 3: Limited Risk. ChatGPT is a generative AI chatbot, which falls under the limited risk category requiring transparency obligations so that users are aware they are interacting with AI-generated content.
The AI system was used by peer reviewers to write and generate the text of their assessment reports.
The reports are clear, from reputable sources, and contain direct quotes and evidence of the incident, such as the 'Regenerate response' button text appearing in the review.
None
Technology
Availability of ChatGPT
Easy access to generative AI tools allowed rapid text generation.
Lack of AI Detection Tools
No automated systems were used to identify AI-generated review text.
Data Inputs
Confidential Data Uploads
Reviewers input confidential grant proposal text into external AI tools.
Human Factors
Academic Overwork
High pressure and lack of time led reviewers to cut corners using AI.
Reviewer Academic Misconduct
Dishonorable behavior where reviewers relied on AI rather than expert analysis.
Process and Methods
Excessive Proposal Length
Proposals of 100-150 pages created massive reading workloads for reviewers.
Inadequate Quality Control
No internal checks to filter out generic or AI-generated feedback.
Lack of Score Transparency
Inability for applicants to see scores made detecting AI reviews harder.
Management
Unmanaged Academic Workloads
Universities failed to allocate formal time for peer review activities.
Slow Policy Updates
Management delayed issuing clear guidelines on generative AI usage.
Failure to Pre-empt Risks
ARC management did not anticipate AI risks despite community warnings.
Regulatory Environment
Absence of Explicit AI Rules
Existing policies did not explicitly mention or ban generative AI tools.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
