Large Language Models (LLMs) like ChatGPT have become widely accepted in various sectors, making it increasingly challenging to differentiate AI-generated content from human-written material. This has raised concerns in scientific research and media, where undetectable AI-generated texts can potentially introduce false information. Studies show that human ability to identify AI-generated content is barely better than random guessing. Moreover, biases can subtly and passively be amplified by LLMs’ consistent output.
To address these challenges, the “distributional GPT quantification” method is suggested. This approach calculates the proportion of AI-generated content in a large data set without examining individual instances. It uses a maximum likelihood estimation for texts of an unclear origin with reference texts known to be human or AI-created, reducing estimation errors and improving computational efficiency.
Empirically, AI-generated texts frequently use certain adjectives, verbs, non-technical nouns, and adverbs more than human-written content, as seen in recent reviews of scientific literature. Researchers can harness this difference to deliver consistent and noticeable results by parameterizing their framework for probability distribution.
The proposed model was tested on a large corpus of reviews submitted to well-regarded AI conferences and publications. The results indicated a small but significant percentage of reviews posted after the release of ChatGPT might have been substantially altered by AI. However, reviews submitted to Nature’s family of publications did not exhibit this trend.
The Stanford research team’s contributions include developing a simple and effective method for estimating the percentage of AI-written or altered text in large data sets, employing a methodology to examine reviews submitted to esteemed scientific and machine learning conferences and observing changes at the corpus level due to integrating AI-generated texts into information ecosystems.
In conclusion, the study suggests a new paradigm for effectively monitoring AI-altered content, emphasizing the importance of broader analysis of LLM output to discern the subtle but lasting effects of AI-generated language on the field of scientific research and publications. All credit for this research goes to the researchers of the project.
