9.1 C
New Delhi
Thursday, December 12, 2024

Generative AI Text Revealed by Specific Keywords

More from Author

In Short:

Researchers have developed a new method to estimate the usage of large language models (LLMs) in scientific writing. By analyzing excess word usage in abstracts from 2023 and 2024, they found that at least 10 percent of abstracts were processed with LLMs. The study compared word frequencies before and after LLMs were widely used, showing a significant increase in style words like verbs and adjectives. This method can help identify LLM-assisted writing.


Researchers Develop Method to Detect LLM Usage in Scientific Writing

AI companies have struggled to detect when a piece of writing was generated using a large language model (LLM). However, a group of researchers has created a new method to estimate LLM usage in scientific writing by identifying “excess words” that appeared more frequently during the LLM era in 2023 and 2024. According to the researchers, at least 10 percent of abstracts in 2024 were processed with LLMs.

Insights from the Study

In a recent preprint paper, researchers from Germany’s University of Tübingen and Northwestern University were inspired by the impact of the Covid-19 pandemic on excess deaths. They analyzed excess word usage after the wide availability of LLM writing tools in late 2022. The researchers discovered an increase in the frequency of certain style words that was unprecedented in both quality and quantity.

Delving Into the Data

The researchers examined 14 million paper abstracts published on PubMed between 2010 and 2024 to track changes in word frequency. They found that certain words significantly increased in popularity after the introduction of LLMs in 2023, such as “delves,” “showcasing,” and “underscores.”

Moreover, the researchers observed a surge in the usage of words like “potential,” “findings,” and “crucial” in post-LLM abstracts. These changes in word use were distinct from previous trends related to major world health events.

An Intricate Interplay

The study highlighted hundreds of “marker words” that became more common in post-LLM scientific writing. By analyzing these markers, researchers estimated that at least 10 percent of papers in the PubMed corpus post-2022 were written with LLM assistance. The actual number could be higher due to potential abstracts lacking identified marker words for LLM usage.

- Advertisement -spot_img

More articles

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.

- Advertisement -spot_img

Latest article