This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The smart contract bug bounty platform Immunefi banned 15 users for submitting bug reports generated by ChatGPT. The platform stated that these reports, while appearing well-structured, were nonsensical and wasted the time of developers and reviewers. Immunefi noted that they will continue to monitor for AI-generated content until such tools are capable of producing accurate reports.
ChatGPT-generated responses submitted to smart contract bug bounty platform Immunefi reportedly lacked details to help diagnose technical issues, which reportedly wasted the platform's time, prompting bans to submitters.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Using AI systems to gain a personal advantage over others such as through cheating, fraud, scams, blackmail or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism for research or education, impersonating a trusted or fake individual for illegitimate financial benefit, or creating humiliating or sexual imagery.
Also classified as
The AI system generated highly polished but technically nonsensical reports, demonstrating a lack of capability to perform the task accurately.
The 15 banned users used ChatGPT to generate fake bug reports in an attempt to cheat the Immunefi bug bounty system for financial gain.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by human users intentionally using ChatGPT to generate and submit fake bug reports to a bounty platform for personal gain.
Added categories peak at 2: Minor (Epistemic Harm, Inferred)
Immunefi banned 15 users for submitting ChatGPT-generated bug reports that were technically nonsensical but polished. This incident represents low-level AI-enabled spam wasting developer resources on a private platform, posing negligible threats to national security.
The report describes ChatGPT as a language tool that converses with humans and answers questions based on intent, acting as an assistant to generate text based on user prompts.
Limited Risk: ChatGPT is a chatbot and generative AI tool, which falls under the transparency obligations of Limited Risk systems under the EU AI Act.
The AI system was used by individuals to draft and generate structured bug reports based on prompts, acting as a writing assistant.
The reports provide clear, consistent details about the incident, including the platform involved, the number of banned users, the reason for the ban, and the specific limitations of the AI tool.
None
Technology
Plausible but Inaccurate Output
Polished presentation successfully disguised technically incorrect answers.
Lack of Technical Reasoning
The tool has no real capability to identify software bugs.
Data Inputs
Biased Training Data
Biased training data causes answers to lack common sense and context.
Human Factors
Overreliance on AI Tools
Users submitted AI-generated reports without verifying their accuracy.
Financial Incentive Exploitation
Users tried to exploit the bounty system by spamming automated reports.
Process and Methods
Absence of Pre-submission Validation
No manual review of AI output was done by the submitters before reporting.
Management
Lack of Clear AI Usage Guidelines
Platform had to reactively ban users due to lack of upfront AI policies.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
