Ensuring Research Integrity: Exploring Tools to Investigate Scientific Misconduct

What is Research Integrity?

Research integrity (RI) refers to the use of honest and verifiable methodologies throughout the proposal, evaluation, and execution of the research study. Once the study is completed, the findings should be reported in accordance with relevant regulations, guidelines, and accepted professional norms. RI emphasizes honesty and accuracy in research, which are crucial for maintaining trust within the scientific community and the lay public and furthering scientific understanding.

Thus, the importance of RI cannot be understated. For journals that publish research with the potential to shape health outcomes, RI protects patients and the public from negative consequences of misleading data. A breakdown of the trust in RI can threaten adoption of public health directives and of medical advice.

Violating RI leads to research misconduct, which is often defined as “falsification, fabrication and plagiarism” of research; however, this definition does not capture the full scope of misconduct seen today. Additional activities such as undisclosed conflicts of interest, violations of research policies, violations of ethics committee regulations, and fraudulent peer review can all undermine faith in RI.

Consequences of Scientific Misconduct

The consequences of scientific misconduct range from the broader social costs to the direct financial burdens borne by stakeholders. On an individual level, research misconduct leads to wasted efforts by researchers who rely on fabricated data, reputational damage to innocent collaborators (such as co-authors, students, and post-docs working under supervision), and significant time and energy spent on investigating misconduct at institutional, academic, and journal levels. Additionally, research funding may be wasted on poorly executed studies that could have been better allocated to research with greater impact, representing a significant loss for both funders and taxpayers

Broader societal costs are evident in cases like Andrew Wakefield’s claims of a possible link between the MMR (measles, mumps, and rubella) vaccine and autism. His conclusions, based on cherry-picked results, continues to fuel public controversy over vaccine hesitancy today.

Scientific misconduct has become increasingly prevalent. A recent study by Freijedo-Farinas et al reports that retraction rates for biomedical papers in Europe have increased fourfold between 2000 and 2021. While research misconduct is not responsible for all these retractions, approximately 67% were due to misconduct. The rise of research misconduct detection tools over the past decade may have contributed to the increase in cases, as these tools enable more effective detection.  However, despite their continued use, the incidents of plagiarism cases have not declined, indicating an increase in these types of issues.

Journals’ and Editors’ Roles in Combating Scientific Misconduct

Journals employ several strategies to catch papers that include scientific misconduct through tightening publication policies, use of tools to identify fraudulent papers, and growing teams of RI specialists.

Elsevier, recognizing the costs of fraudulent research, has significantly expanded their RI team from two to more than twenty members in just a few years. Several other big publishers such as SAGE and Wiley have publicized their RI groups, demonstrating a shift  in the publishing industry toward ensuring high ethical standards in scientific articles.

Scientific journals are essential for the dissemination of scientific data, and journal editors play a key role in maintaining the integrity of the papers they publish. The development and use of sophisticated tools for detecting and preventing research misconduct are important for preserving the integrity of scientific research. As the publishing industry adopts these technologies and expands its research integrity teams, these efforts will help enhance the credibility and reliability of scientific knowledge.

Tools to Protect Research Integrity

The following section explores various tools developed to aid in the detection and prevention of research misconduct. These tools address different facets for research misconduct from papermill products to systematic manipulation of peer review.

Image Manipulation

imageTwin – imageTwin is a software solution to identify inappropriate image manipulation within individual papers and across publications. Scanned figures are compared against one another for duplication and indexed against a database of over 50 million images to detect plagiarism across publications. While imageTwin effectively detects manipulation and duplication across various figure types (including western blots and light photography), it is especially powerful at detecting duplication within microscopy images. The tool generates an easy-to-understand report, providing publishers with a “first order” examination for further investigation.

Proofig – Using AI and computer vision, this software identifies duplications in various scientific images, including western blot bands, microscopy images, flow cytometry, plates, mice, in vitro, and in vivo figures. Proofig can identify manipulations such as rotations, scaling, flipping, cropping, and both full and partial duplications within the paper that it is scanning. A report is then generated for the user, where validation of the results is required to ensure the accuracy of the identified issues and eliminate any false positives.

FigCheck – FigCheck is an image checking tool spun from a project of Tongji University’s “venture valley” initiative. Like Proofig and imageTwin, the tool uses AI algorithms to detect manipulations like rotations, scales and crops; and generates a report for users to verify. Users are able to test the power of Figcheck before fully committing to the tool, as the platform offers a free, 1-time-per-week usage.

ImaCheck – ImaCheck has functions similar to the tools discussed above and is available as a web-based and a locally hosted solution. Complemented by an API tool, this enables Imacheck to integrate with current software used by the publisher. Imacheck has a built-in comparison tool feature, which allows users to validate issues identified by Imacheck without having to switch to another platform.

Adobe Photoshop^® – Adobe Photoshop might be the most commonly used and oldest tool in the industry when it comes to investigating image integrity. The investigatory process using Adobe Photoshop is manual, and a complement to validating issues identified using an automated platform like those described above. The U.S. Office of Research Integrity^® (ORI) has released two Forensic Actions tools (Basic Forensic Actions and Advanced Forensic Actions) for Adobe Photoshop, to help with the visualization of image inconsistencies during a forensic examination. Additionally, the ORI has also released samples from closed ORI integrity cases to illustrate the different degrees of image manipulation in biomedical sciences.

Papermill Detection

STM’s Papermill Detection Tool – The Papermill Detection Tool was launched by the operational arm of STM in April 2023, and it is a stand-alone web application where manuscripts submitted to the journal can be uploaded and checked for a variety of signals indicative to papermills. Since the release of the minimal viable product, it was quickly adopted by major publishers such as Taylor & Francis and Springer Nature to name two. While the exact signals in the Papermill Detection Tool remain proprietary, in September 2023, STM Solutions announced their  joint efforts with Clear Skies (described below) to incorporate the Clear Skies Papermill Alarm as an alert within the tool.

Clear Skies – Papermill Alarm – Developed by Adam Day, the Clear Skies Papermill Alarm was the first tool to perform papermill screening across the entire literature. It is an early-warning system that uses a simple traffic light system to alert publishers to risk of papermill activity. Papermill Alarm uses network analysis and AI to detect signs of organised research fraud. A red alert indicates a high-similarity rate to papermill products, and a green rating indicates no resemblance to such papers.

Problematic Papermill Screener (PPS) – Tortured phrases can be considered a ‘fingerprint’ for detecting papermill activity. They can be described as unusual expressions or phrases in lieu of well-established scientific terms. For instance, ‘counterfeit neural organization’ in lieu of ‘artificial neural networks’ is a tortured phrase identified on a published article flagged on PubPeer. These phenomena can arise as a result of machine paraphrasing to avoid plagiarism detection.

PPS is a website that tracks papers containing tortured phrases. In a published pre-print, as of 13 October 2023, PPS has reportedly flagged 11,945 papers containing tortured phrases with the help of scientific sleuths. Problematic papers are reported on PubPeer under the Tortured Phrase Collection. This collection is particularly useful for detecting trends of tortured phrases during integrity investigations and for determining if the paper under scrutiny has been flagged by the PPS.

Signals – Along similar lines, Signals is an app-based tool aimed at identifying fake or fraudulent research, including papermills. The tool offers two dimensions of protection: i) at the manuscript level and ii) at the journal level. The manuscript metadata is scanned for flagged authors, level of self-cited articles, citation of retracted articles, and information on the institutional research output that the authors are affiliated with. Taken together, Signals generates an article risk level that can provide an indication of whether a full investigation is required. At the journal level, Signals is able to provide an all-time journal evaluation reflective of the risk level of individual articles that were published. An example dashboard of their app illustrates how the tool works.

Peer Review Manipulation

Unsual Activity Detection Tool – Embedded within ScholarOne, Clarivate developed the Unusual Activity Detection (UAD) Tool to detect fraudulent peer review activity using a proprietary algorithm. The tool provides journals with a risk indicator on manuscripts using web traffic and server data. In a discussion forum held by COPE, they highlight the tool’s ability to detect if submissions are from the same computer. Multiple submissions from a single computer may be indicative of papermill submissions and a signal to initiate investigation on the submission. Publishers such as the Royal Society of Chemistry use the UAD tool as a first indicator for papermill investigations.

Incorrect Nucleotide Sequence Reagent Detection

Seek & Blastn an open-access, online, web-based platform that allows users to fact-check nucleotide sequences reported in the manuscript. Once a PDF is uploaded into the platform, sentences containing nucleotide sequences are automatically analyzed to determine whether the gene it targets is correctly identified. The current version of the tool fact-checks nucleotide sequences against the NCBI human genomic and transcript database. This functionality allows the tool to identify nucleotide sequence reagent errors in scientific papers. Reporting of incorrect nucleotide sequences leads to irreproducible research results. While, in most cases this may be due to unintentional errors, in others, this may be fraudulent research. The development and validation of the Seek & Blastn tool is discussed in this paper published by PLOS ONE.

Bibliography Check

Scite.AI – Scite.AI a next-generation tool for citation analysis using Smart Citations. A solution that the platform offers for publishers is Scite’s Reference Check. This helps users to understand the quality of citations in the article and is a means to quickly identify if a cited article has been retracted, disputed,or is out-of-date. The tool shows how the citation was used within the article; thus, it is a useful modality to identify inappropriate citations.

Irrelevant/inappropriate citations are a hallmark of papermills, the use of Reference Check can help detect fabricated or low-quality research through the identification of irrelevant/inappropriate citations. By analyzing citation patterns and the context in which citations are made, publishers can better identify and scrutinize low quality papers more effectively.

Conclusion

A spectrum of RI tools is available on the market to help combat scientific misconduct. These tools can detect image manipulation, unusual citations and erroneous nucleotide sequences, and alert to potential papermill submissions.

For publishers or journals looking to integrate RI tools into their daily operations, consulting with the provider and trialing the product can be the first step to understanding how the tool works. Special interest working groups also exist where research integrity specialists discuss and share valuable insights and tips from their experiences with the tools. Engaging with these communities can provide publishers and journals with innovative strategies to maximize the value of these tools.

It’s important to consider that while these tools are invaluable in identifying signs of manipulation, they should not replace the human component of the investigation. Tools are extremely useful for flagging suspicious papers, but the outcome of the investigation is ultimately determined by the investigator. COPE shares guidelines and workflows for investigating different types of scientific misconduct, but detailed standard industry protocols are still lacking. The integration of advanced RI tools with robust human oversight is essential for maintaining the integrity of scientific research. By combining technological advancements with expert judgment and clear guidelines, the scientific community can more effectively address and prevent misconduct, ensuring the reliability and credibility of published research.

Finally, we’d love to hear from you. Have you used any of these tools? If so, what was your experience? Or are there other tools you’ve found helpful that we haven’t mentioned? Share your insights and suggestions with us to contribute to a broader conversation.

Conflicts of Interest:

None to disclose

Origin Editorial is now part of KnowledgeWorks Global Ltd., the industry leader in editorial, production, online hosting, and transformative services for every stage of the content lifecycle. We are your source for society services, market analysis, intelligent automation, digital delivery, and more. Email us at info@kwglobal.com.