The use of large language models (LLMs), such as ChatGPT, has significantly increased in academic writing, resulting in observable shifts in writing style and vocabulary, particularly in biomedical research. Concerns have risen around the authenticity and originality of scientific content and its implications for research integrity and the evaluation of academic contributions.
Traditional methods for detecting LLM-generated text in literature involve LLM detectors trained to distinguish between human and AI-generated text, modeling word frequency distributions, or tracking the frequency of marker words overused by LLMs. To overcome the limitations of these methods, a new data-driven approach has been proposed which relies on tracking the usage of certain words for a comprehensive analysis of the impact of LLMs on scientific writing.
In a study spanning from 2010 to 2024 that analyzed over 14 million PubMed abstracts, authors identified words whose usage showed significant increase post-ChatGPT release – these were dubbed “excess words”. The frequency of these words was compared to projections from previous years to calculate the “excess frequency gap” (difference between observed and expected frequencies) and the “excess frequency ratio” (ratio of observed to expected frequencies). The study revealed that the frequency of certain stylistic words such as “delves,” “showcasing,” and “underscores” marked an unprecedented rise post-ChatGPT release, hinting LLM involvement.
The researchers used this method to establish a lower limit for LLM influence. To illustrate, the word “potential” exhibited an excess frequency gap implying LLM was responsible for its usage in at least 4% of the 2024 abstracts. By this analysis, the authors concluded that a minimum of 10% of papers in 2024 were written with LLM assistance. This figure differed across disciplines, countries, and journals with the highest percentage of LLM usage (35%) found in computational papers from China. However, the actual extent of LLM influence may be higher as some LLM-generated abstracts may not contain any of the tracked “excess words”.
The study unearths substantial change in academic writing style due to LLMs such as ChatGPT. At least 10% of recent biomedical abstracts show signs of AI assistance, illustrating the transformative impact of LLMs on scholarly communication. This raises critical questions about research integrity and the future of academic writing.
