Top studies about ChatGPT in education have been removed from red flags

Research that said OpenAI’s ChatGPT can affect student learning has been return about a year after publication. The journal’s publisher, Springer Nature, cited “inconsistencies” in the analysis and a lack of confidence in its results — but not before the paper was cited in hundreds and circulated on social media.

“The authors of this paper also reported some interesting findings about the benefits of ChatGPT on educational outcomes,” he said. Ben Williamsonsenior lecturer at the Center for Research in Digital Education and the Edinburgh Futures Institute at the University of Edinburgh in Scotland, in an email to Ars. “It was seen by many on social media as one of the first pieces of hard, gold proof that ChatGPT, with high-output AI, benefits students.”

The made paper tried to quantify the “effects of ChatGPT on students’ learning, learning style, and critical thinking” by analyzing the results of 51 recent studies. His meta-analysis calculated the size of the results between different experimental groups that used ChatGPT in educational groups and controls that did not use an AI chatbot.

This analysis is supposed to show how “ChatGPT has significant benefits in improving performance” and “helps in the development of the learning mind” and “enhances higher thinking,” according to the researchers who wrote the paper. The results now returned appeared first in the magazine Humanities & Social Sciences Communications, published by Springer Nature on May 6, 2025.

“Sometimes they seem to be combining the best studies, or combining findings from studies that may not be comparable because of methods, populations and samples,” Williamson told Ars. “It looked like a paper that shouldn’t have been published in the first place.”

Williamson also questioned the timing of the paper’s publication two and a half years after OpenAI was released ChatGPT in November 2022. “It is unlikely that many of the high-quality studies related to ChatGPT and the learning process could have been conducted, reviewed, and published in that time,” Williamson said.

A legacy that can continue to repay

Since its publication, the study has been cited 262 times in other papers published by the peer-reviewed journal Springer Nature and received full acceptance. 504 words from peer-reviewed and non-peer-reviewed sources. It also attracted nearly half a million readers and was well received interest in the Internet ranking in the 99th percentile of newspaper articles in terms of interest.