This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A professor at Texas A&M University-Commerce incorrectly used ChatGPT to detect AI-generated content in student essays, resulting in false accusations of cheating and the temporary withholding of diplomas for several seniors. The professor mistakenly believed ChatGPT could identify its own output, despite the model not being designed for such detection. The university subsequently investigated the incident, confirmed that no students were ultimately barred from graduating, and began developing policies regarding AI usage in the classroom.
A Texas A&M-Commerce professor reportedly informed his class of his misuse of ChatGPT to detect whether student submissions had been generated by the chatbot itself, which informed their graduation status.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
Also classified as
ChatGPT failed to perform reliably by hallucinating and falsely claiming authorship of the students' essays and other unrelated texts.
The professor placed inappropriate trust and reliance on ChatGPT to perform academic plagiarism detection, a critical task the system was never designed or fit to carry out.
Human
Due to a decision or action made by humans
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was primarily caused by the professor's human decision to misuse ChatGPT as an AI detector, which led to unintended academic distress and false accusations.
Added categories peak at 2: Minor (Psychological Harm, Indirect)
A university professor falsely accused students of cheating by incorrectly relying on ChatGPT to detect AI-generated content. The incident represents a local human-computer interaction and academic policy failure with negligible implications for national security.
ChatGPT acted as an assistant by responding to the professor's queries and providing information regarding the text's origin, which the professor then used to make grading decisions.
Risk Level 2 (High Risk): The AI system was used in an educational setting to evaluate students and make decisions that directly affected their academic progression and graduation.
ChatGPT is a general-purpose conversational LLM designed to function as a chatbot and writing assistant, though in this incident it was inappropriately used by a human for cheating detection.
The incident is documented across multiple reputable news sources with consistent facts, including statements from the university spokesperson confirming the key details of the grading dispute and the professor's actions.
None. The facts of the professor's misuse of the tool and the subsequent resolution are clear.
Technology
ChatGPT False Affirmation
ChatGPT falsely claimed authorship of the students' submitted essays.
Lack of Built-in Detection
ChatGPT is not designed or capable of detecting AI-generated text.
Data Inputs
Unverified Bot Responses
Professor relied on unverified ChatGPT outputs as proof of cheating.
Ignored Google Docs Timestamps
Professor initially ignored student-provided Google Docs timestamp evidence.
Human Factors
Professor Lack of AI Literacy
Professor did not understand how ChatGPT works, misnaming it Chat GTP.
Confirmation Bias
Professor assumed guilt and accepted chatbot claims without verification.
Student Distress and Panic
Students experienced severe distress due to false accusations of cheating.
Process and Methods
Flawed AI Detection Method
Professor copied essays directly into ChatGPT to ask if it wrote them.
Lack of Standardized Review
No formal process existed for verifying academic dishonesty involving AI.
Management
Inadequate Faculty Training
University failed to train educators on AI capabilities and limitations.
Delayed Administrative Response
Initial lack of clear guidance from administrators when students appealed.
Regulatory Environment
Absence of AI Classroom Policies
The university lacked clear regulations and guidelines on AI use.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
