AI detectors are ‘mislabeling students’ work,’ leading to false accusations

Some AI detectors are reportedly falsely accusing students of using artificial intelligence to submit assignments and exam papers. According to emails seen by Bloomberg Businessweek, a student who has autism spectrum disorder said she tended to write in a formulaic manner, which was misconstrued as being AI-generated. Consequently, she was given a written warning about allegedly plagiarizing work.

Bloomberg conducted a test on 500 college applications submitted to Texas A&M University before the release of ChatGPT, using AI detection services like Copyleaks and GPTZero.

The services flagged 1-2% of these applications as “likely” written by AI, with some being labeled with 100% certainty. The accuracy of AI detectors has been questioned, as many of these applications were written by humans and not part of any dataset used to train language models.

Though the test sample was small, it points out a broader issue. AI detection tools are not foolproof and can wrongly flag human-generated content as AI-produced. It also poses potential risks for students, who may face serious consequences if their work is mistakenly flagged by such systems.

For example, a 2023 Stanford University study found that AI detectors had flagged more than half of the essays written by non-native English students as AI-generated. Ethan Mollick, University of Pennsylvania’s Wharton management professor, took to X to criticize the tools. He said: “It is morally wrong to use AI detectors when they produce false positives that smear students in ways that hurt them and where they can never prove their innocence.”

It is morally wrong to use AI detectors when they produce false positives that smear students in ways that hurt them and where they can never prove their innocence.

Do not use them. https://t.co/cSBZRHgif5 pic.twitter.com/47lFXqT4G7

— Ethan Mollick (@emollick) October 18, 2024

The report also cites the Copyleaks co-founder, who acknowledged that no detection system is 100% accurate and should be used to spot trends, not definitive answers. Alon Yamin added: “We’re making it very clear to the academic institutions that nothing is 100% and that it should be used to identify trends in students’ work.

“Kind of like a yellow flag for them to look into and use as an opportunity to speak to the students.”

Similarly, GPTZero’s co-founder, Edward Tian, admitted that AI detectors have “blind spots,” adding that his company has made strides in debiasing results for ESL students in particular.

In February, ReadWrite reported that a survey has found more than half of undergraduate students in the United Kingdom were turning to AI to complete their studies.

OpenAI also recently stated that it has held back from releasing an AI writing detection tool, citing concerns that it could disproportionately impact certain groups, such as ESL students.

ReadWrite has reached out to Copyleaks and GPTZero for comment.

Alon Yamin, CEO and co-founder of Copyleaks told us, “At Copyleaks, we understand the profound impact false accusations can have on a student’s academic career, and we take on this responsibility with the utmost seriousness. While our AI Detector is over 99% accurate with a market-low false positive rate of 0.6%, no technology is perfect, no matter your industry. In fact, many other services used on a daily basis have higher false positive rates – from technology for medical diagnoses, pharmaceuticals, and credit score models to navigation systems – yet we still use and trust them.

“While the Bloomberg test found a 1% to 2% false positive rate in detecting AI-generated text, it’s important to note that no detection tool is infallible, especially when working with pre-ChatGPT content. We’re continuously improving our accuracy through machine learning. Nevertheless, no matter how accurate any tool may be, we will always encourage educators to use these tools as a guide rather than a definitive tool for policing and judgment. Our goal is to foster open dialogue between teachers and students when potential AI use is flagged, offering an opportunity to understand students’ work and learning processes better. AI detection should be a tool for growth, not a barrier to it, ensuring fairness and transparency in the academic process.”

Featured image: Canva 

The post AI detectors are ‘mislabeling students’ work,’ leading to false accusations appeared first on ReadWrite.