Making “Pay Peer Reviewers” More Than a Slogan

First posted on the ORIGINal Thoughts Blog

By Zen Faulkes
Germanna Community College

LinkedIn: https://www.linkedin.com/in/doctorzen/

Bluesky: @doctorzen.net

Take Home Points:

Peer review is one of the defining characteristics of modern academic writing. Peer review has always been contentious in both concept and practice, but professional research careers have long depended on peer reviewed publications. Because peer review is part of how researchers are hired, fired, and promoted, changing peer review has real consequences for researchers and cannot be done lightly.

Peer review takes many forms, but people usually think of peer review as having these features:

1. It occurs before the work has been widely shared in public.
2. It is organized by independent editors, usually of journals.
3. The reviewers are experts who volunteer their time to read the paper and write the review.

The last point, that review is provided as a service to others, is the focus of this post.

The Reviewer Availability Problem

The number of academic publications has been rising for decades, but the number of prospective reviewers has not kept pace, creating pressure on the peer review practices that have been common for decades. Finding willing peer reviewers has been a perennial complaint of editors and associate editors, but this became harder during the COVID pandemic and stayed that way.

For example, David Shiffman, a journal editor, expressed his frustration with securing reviewers, in a social media post.  Finding two people to agree to review a paper often requires asking 20 or 30 people, with many—or, in some cases, almost all—never even bothering to click a button to decline the review. “I’m not blaming anyone individually here for what I’m sure are 22 reasonable and defensible choices to not help here,” Shiffman wrote. “But in aggregate, it’s a big problem.”

What are journals to do when faced with a disengaged community? Some journals occasionally give up, and “reject” a paper because they couldn’t find anyone willing to review it. But most editors continue searching for potential reviewers, which delays publication. These delays slow the spread of scientific knowledge, but people are often more concerned with the effects on individuals: delays in publication can mean the difference between getting job offers, being awarded grants, and receiving promotions.

Given this shortfall in reviews being written, it’s reasonable to ask if there are incentives that could increase the number of reviewers agreeing to write reviews. Many people have suggested the most widely used incentive in developed economies: money. That is, paying researchers for performing peer review.

The Unpaid Labor Argument

Part of the appeal of paying peer reviewers is that academics see scholarly publishing as somewhere between “unfair” and “exploitative.” Most academic papers in the English-speaking world are published by one of five large and extremely profitable academic publishers.

At a time when academics are asked to compete more to chase grant dollars and other limited resources, seeing publishers earn money from their work is upsetting. This is understandable, given that in much of the world, salaries are relatively stagnant compared to the costs of living and housing. Many people are sensitive to what work they are asked to do and how much they are compensated for that work.

Academics are increasingly describing peer review as labor that should be compensated rather than service to their community. But the service model of peer review has one great benefit: simplicity. People volunteer to write reviews, or they do not.

A compensated model of peer review will be far more complicated. The most basic question is, how much should someone get paid for peer review?

Proponents of paid peer review have rarely put forth detailed proposals for what compensation should look like. In an article on the money value of peer review, Sergio Copiello estimated the value of peer review at somewhere between €50-92 ($54-98) . James Heathers, a researcher who joined a tech startup, suggested a contractual flat fee of US$450. To give one example of how this level of compensation could affect some researchers, Shiffman wrote that if he were paid $450 for every peer review he did in 2023, it would have more than doubled his income for the year.

The pay scale is only the start of questions around compensation. What work is worth being compensated for? What would be the terms and conditions for compensation? What happens when the work is inadequate? Who is invited to review and thus eligible for compensation?

A contractual model raises more questions. Would institutions expect their legal team to review every contract? Many institutions have rules about performing work for other employers. Where would these contracts fit under typical rules?

As the saying goes, the devil is in the details. Proponents of paid peer review would do well to develop detailed proposals that address questions like these. Groups that specialize in metascience and research culture could develop detailed white papers that might articulate some of these questions and gather input from stakeholders about how different scenarios would play out.

Empirical Tests

Ultimately, whether journals should pay peer reviewers has two elements to it. There’s the principle of the thing, but there’s also an empirical question: can paying peer reviewers result in the desired effects (increased participation and timely reviews) while avoiding obvious pitfalls (poorly written and unhelpful reviews)?

Three studies have tested this: one in economics, one in medicine, and one in biology. All three journals offered less than Heathers’ suggested USD$450. The highest pay was less than US$300.

All of these pilot studies found that paying for peer review improved the timeliness of peer review without reducing the quality of reviews received. That’s an encouraging proof of concept.

There are two things to keep in mind in reviewing these studies.

The first is the balance of costs and benefits. In one test, payment resulted in an 8% higher rate of reviews being submitted one day earlier. Such modest improvements may not warrant the financial costs to journals. Moreover, time spent reviewing contracts and managing paperwork might negate the gains made by faster writing of reviews.

The second is that there is a big difference between a small pilot and systematic adoption of payment. Reviewers are operating in an environment where payment is not expected. What happens when payment is expected across the board? Things that are not problems in a small trial could become problems if payment becomes institutionalized.

Author Acceptance

Discussions around peer review focus on publishers, who would have to pay and are reluctant to do so, and reviewers, who would benefit from being paid. But there are at least three parties involved in peer review: publishers, reviewers, and authors.

Most authors also serve as reviewers and vice versa, but the psychology of writing a review and being reviewed are very different. Some researchers might be fine with a journal paying for reviews until the precise moment they get an unfavorable one. Then the questions begin about whether the reviewer just agreed to review for the money, and so on.

If authors believe that payment compromises the reviews, journals would be facing not only increased costs of paying reviewers, but reduced numbers of article submissions and potentially a decline in reputation.

More surveys on the attitudes of submitting authors would be helpful in gauging acceptance or resistance to paid peer review.

Expect More Debate

The problems that bedevil peer review are not unique. There is another system that relies entirely on some people willingly contributing to benefit others in their community: blood supply.

Blood transfusion is lifesaving, and there is no substitute for blood. Globally, voluntary and uncompensated donations are the preferred source of blood, but some argue that the medical need for available blood is so great that incentives to donate blood are somewhere between “acceptable” and “necessary.”

The research on incentives for blood donation paints a complex picture. In most countries, blood donation is unpaid, but different countries have different policies around incentives. Likewise, the attitudes of donors towards incentives vary widely across countries. What incentives work best and what is a reasonable or fair incentive is also studied, including whether incentives other than direct payment can be as effective as cash.

Blood donation has been routine for decades, and the stakes are higher than in peer review. That there is not a clear answer in this field—after all this time—suggests that we should not expect discussions around paid peer review to conclude any time soon.

The Road Forward

Here are several suggestions for how the research community might make progress on whether peer review should be paid or not.

First, proponents of paid peer review should start developing suggestions in more detail, addressing the sorts of questions laid out here. Frequently, what seems to be an obvious solution to one person is not obvious to their colleagues, and it is likely that more consciousness raising among academics would be necessary for paid peer review to be more than a hypothetical or pilot study.

Second, many researchers may find data more convincing than arguments from first principles. More pilot studies from journals would be extremely welcome. These could help to force people to grapple with the details of implementation and determine what the cost-benefit ratio might be.

Third, organizations that specialize in research culture, metascience, and scholarly publishing are well-placed to examine the topic. Reports or white papers on the topic could help articulate the range of suggestions, and their potential pitfalls, from the perspective of all of the players.