This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
AI writing detection tools, including those from Turnitin, GPTZero, and Copyleaks, are frequently misidentifying human-written student work as AI-generated. This issue disproportionately affects neurodivergent and ESL students, leading to academic penalties, probation, and significant student anxiety. While developers claim high accuracy, independent testing and academic studies confirm significant false positive rates, creating an environment of paranoia in educational settings.
AI writing detection tools have reportedly continued to falsely flag genuine student work as AI-generated, disproportionately impacting ESL and neurodivergent students. Specific cases include Moira Olmsted, Ken Sahib, and Marley Stevens, who were penalized despite writing their work independently. Such tools reportedly exhibit biases, leading to academic penalties, probation, and strained teacher-student relationships.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
Educators overrely on AI detectors to make critical disciplinary decisions, treating them as definitive proof of cheating.
The AI detection tools fail to perform reliably, exhibiting a 1% to 4% false positive rate on human-written text.
The AI detection systems perform with significantly lower accuracy for ESL and neurodivergent students, falsely flagging their writing as AI-generated due to their formulaic or non-native writing styles.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The false accusations of cheating are caused by the incorrect classifications (false positives) generated by deployed AI writing detection systems, which is an unintentional failure of their design.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
AI writing detection tools are generating high rates of false positives, disproportionately penalizing ESL and neurodivergent students in US universities. While this raises minor concerns regarding algorithmic discrimination and student rights, it represents negligible threat to core national security interests.
The AI systems (Turnitin, GPTZero, Copyleaks) operate as tools that perform a specific, well-defined function: analyzing text inputs to output a probability score of AI generation. They do not exhibit goal-directed planning or operate independently in real-time environments.
High Risk: The report describes AI systems used in educational settings that directly affect students' grades, academic standing, and access to education, which aligns with the High Risk category for educational and vocational training systems.
The primary function of Turnitin, GPTZero, and Copyleaks in this context is to detect academic cheating by identifying AI-generated text. Content verification is a secondary goal as they verify the origin of the written content.
The report provides extensive, high-quality information including specific student case studies, independent testing data from Bloomberg, academic studies from Stanford, and statements from both AI developers and educators. The causal links between the AI tool failures and the resulting academic and psychological harms are clearly documented.
One could argue the primary failure is human overreliance by teachers rather than the AI's technical limitations, but both aspects are thoroughly covered.
Technology
Inherent False Positive Rates
Detectors have a 1-4% false positive rate, leading to incorrect flags.
Perplexity-Based Detection
Detectors flag formulaic writing styles, misidentifying human text as AI.
Grammar Tools Triggering Flags
Using tools like Grammarly to improve text can cause 100% AI-written flags.
Data Inputs
Lack of Neurodivergent Data
Models are not trained on neurodivergent student writing styles.
Biased Training Data
Models struggle with non-native English writing, leading to higher false rates.
Human Factors
Teacher Over-Reliance on Tools
Educators treat detector scores as absolute proof of cheating.
Formulaic Student Writing Style
Neurodivergent and ESL students naturally write in a more structured manner.
Process and Methods
Lack of Clear Disciplinary Path
Schools lack standardized processes to handle and resolve false AI accusations.
No Verification of Writing Steps
Students lack easy ways to prove their authentic drafting and writing process.
Management
Adoption of Unverified Software
Schools purchase and deploy AI detectors without verifying their accuracy rates.
Prioritizing Cheating Prevention
Administrators prioritize catching cheaters over the risk of false accusations.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
