Practical Advice for Implementing Double Anonymous Peer Review: Ask the Editorial Experts

First posted on the ORIGINal Thoughts Blog

Take Home Points:

Double-anonymous peer review (DAPR) refers to a model in which authors as well as reviewers are unknown to one another. It stands in contrast to the single-anonymized process (in which authors are known to reviewers) as well as the emergent models of open and transparent review (where all are known to one another; and, in the case of the latter, elements of the recommendation are published with the article’s version of record). Interest in frameworks beyond single-anonymized has grown in recent years, spurred by calls for innovation, the desire to increase both the integrity of the overall process and quality of submitted reviews, and as corollary to overdue, necessary conversations regarding the redress of systemic biases both within scholarly communication and public life more broadly.

In Part I of this series, Kristen Overstreet provided a considered and comprehensive overview of DAPR and those factors which might favor its adoption. While opinions will continue to diverge on the overall effectiveness of this model (e.g., see Bastian on potentially faulty assumptions; Bauersfeld Et al. on the ability of AI tools to subvert anonymization; Waltman Et al. on the benefits and deficits intrinsic to any chosen approach; etc.), editorial professionals would nonetheless benefit from insights on how best to operationalize DAPR should a journal team be so persuaded. To that end, we asked members of the Origin Editorial team and their colleagues in the field of scholarly publishing with the requisite expertise: What practical advice would you offer for implementing DAPR, either from whole cloth or transitioning from a single-anonymized model?

Christine Beaty
While double-anonymous review is a worthy goal, journal staff’s time and resources are finite. If you’re considering a switch, it’s important to map out potential effects on all stakeholders – editors, authors, reviewers, and staff – to understand where you might encounter pain points and where adjustments may be needed.

Of the American Heart Association’s 14 journals, one currently utilizes DAPR, in a workflow that was implemented in early 2022. A few factors complicated the transition. During the pandemic, when submissions were high and resources were already spread thin, staff did not have the bandwidth to operationalize a total shift in the journal workflow. Further, as part of a portfolio with a robust cascade mechanism, we could not guarantee that all submissions would have a completely double-anonymous experience.

Ultimately, we felt that we could not move the journal to full DAPR; instead, we chose to implement a pilot initiative where we allow authors to choose whether their manuscript is reviewed in a double- or single-anonymous fashion.

For a successful implementation, staff and editors need to be on the same page about how much anonymization is expected and feasible. Our approach is primarily author-driven: staff will perform a basic check (e.g., that a title page has been uploaded as a separate file), but do not scour the manuscript for institution names, phrases indicating self-citation, or other more detailed checks. This is in line with our portfolio’s general approach to minimize friction at initial submission.

On a practical level, be mindful of your submission platform’s capabilities. We are fortunate that our platform allows for this author-choice hybrid model, but that might not be the case universally. During our development stage, we encountered an unexpected hurdle in the form of the auto-generated cover page appended to all our submissions, which by default includes the author list (undoing any effort at anonymization made by the authors!). An enhancement was developed to get past this, but it added some time to our rollout.

Lastly, as you prepare for launch, make sure that your authors know what to expect with DAPR. In addition to clear and easily accessible author instructions, consider some promotional materials, as well as a landing page or blogpost if you have the capability.

Jennifer Mahar 
If you’re considering making the switch to DAPR, it’s important to take a holistic view of your workflow. The practical considerations are:

1. Can your peer review system accommodate a double-anonymous workflow?
2. Can you afford to spend time setting up and managing a new workflow?

If the answer is yes, then you’re well on your way. I currently manage a journal that has a double-anonymous workflow. The issues that I encounter hinge on the two questions above. The system we use is not ideal for double-anonymous – there are manual checks and balances that must be incorporated during my process for every single manuscript. There are also steps that the editors must incorporate during their process, which makes for a challenging day at times. For example, we must queue or ‘pop-up’ certain letters so we can modify them on the fly to remove names and identifiable information. I must be sure that there is nothing identifiable in the manuscript and that there is a separate title page. I also must review the references to be sure there aren’t too many by the author or that they have also been shielded from the eyes of the reviewer. Admittedly, these are all routine occurrences in my editorial office, but if you’re considering a switch, it’s important to take into consideration all the variables and map out your workflow ahead of the change.

The benefits of DAPR absolutely outweigh any of the additional time and effort for my journal. We have extremely sensitive content and a small community; DAPR helps to shield both parties and takes bias off the table, which in the end is paramount to my community.

Christina Nelson 
The Journal of Bone and Joint Surgery has used a fairly rigorous DAPR process for our flagship journal for decades and has extended this model to four of the five newer sister journals (with the exception being our video surgical techniques journal due to the nature of the multimedia content and review process). Although staff do read/skim every manuscript prior to editor assignment, the time commitment is manageable and contributes to a less-biased review experience, especially in a small subset of the medical field such as ours.

If you are considering a DAPR model, any changes to the editorial process benefit from thinking through the details (and knowing that there will always be surprises!). Some questions to ask yourself, the editors, and the team are:

1. How much information will be anonymized? Submission systems can automatically withhold many identifying details, but the manuscript text tends to reveal the authors via institution names, author initials, trial registration numbers, “our previous study” references, etc.
2. Which article types should this apply to? An editorial or invited submission may benefit less from anonymization than unsolicited scientific articles, or a particular article type’s content may be too difficult to anonymize while preserving meaning.
3. Are different processes needed for certain submission items? Figures and tables may require different protocols and software when it comes to downloading, modifying, and re-uploading files (and possibly anonymizing file names, if they are downloadable for reviewers).
4. How will exceptions be handled? Gray areas abound and establishing a chain of internal command for queries can save time. A shared knowledge file also helps to serve as a common reference point.
5. How long will the process take? Timing how long it takes to skim papers and anonymize them helps establish benchmarks. Using shortcuts such as searching for author names or key words can save time.
6. Who should be informed and when? A communication plan that includes timelines for updates to the author instructions, system settings, and messaging to stakeholders offers clarity and reduces confusion.
7. How will users be able to access identifying data if needed? Having a de-anonymized version uploaded as a companion file is helpful for information requests.
8. Is there any way to gauge the impact of these changes? Consider a survey to see how reviewers, editors, and staff feel about the process. Manuscript processing times and number of reviewer/editor complaints are other useful metrics to track.

Kristen Overstreet 
Step 1 for creating a DAPR process or transitioning to one is to answer the question, what is our purpose?  Having a very clear understanding of what you want to achieve with the type of peer review process you choose will help you answer all the other questions along the way.  Why is DAPR better for your journal community?  Might it reduce instances of systemic bias in the peer review process compared to single-anonymous because the reviewers do not receive information about who the authors are?  For example, might your journal editors and reviewers make conscious or unconscious recommendations/decisions based upon gender, race, geography, hierarchy, etc., if they are aware of this information? Are you having trouble securing reviewers, and might DAPR result in higher reviewer acceptance rates if the reviewers know that their identities will not be revealed to the authors, as they would be with an open process?  If you are transitioning, what problem are you trying to solve in your current process?  If you are creating a peer-review process, become familiar with all types of peer review and determine what best fits the needs of your journal and community.

Step 2 is to establish your resources. If you decide DAPR is right for you, be aware that it may require more resources in the editorial office.  Checking in new submissions now includes ensuring anonymity.  How strict you will be in the anonymizing process relates to your purpose in choosing DAPR.  My co-authors of this post have given examples of the differences in their DAPR processes.  Some are very strict; others leave the onus with the authors.  The stricter you are, the more resources you need.  It will take more time to check in each submission.  Additionally, what responsibilities will lie with the editors in this process?  Are they on board with increasing their workloads?

Once you have answered the question of why you are choosing DAPR, you can begin to determine the policies and procedures related to your journal’s implementation of this process. Be sure to capture this in writing so that all involved with the journal in the future will be aware of why the decision was made and can determine how to adapt the implementation, as needed, according to that purpose.

Jason Roberts, PhD
Perhaps by a quirk of fate, I have always run double-anonymized clinical medical journals. In an ideal world, it seems to me journals should either operate under DAPR, to mitigate the potential for bias, or some form of open peer review, for the sake of transparency.

However, I also know there are considerable practical limitations that make it rather impossible to completely mask the identity of authors. Just focusing on the submitted manuscripts, DAPR if done to its full extent requires a lot more than stripping out the title page.

Many authors start a paper by immediately referencing their previous work. So, do you redact those references? But if you do that, does that make the peer reviewers job that bit harder? They may not now be able to properly understand the context for the research. Conversely, if the reviewer is very familiar with the field, they likely will figure out the identity of the authors simply because of the act of redacting the reference. Alternatively, they may be wondering why such an obvious reference was missing (when in fact it was simply redacted).

Where do you stop with the anonymizing process? Do you include or redact the name of the institutional review board? What about the clinical trial registration number? Many journals mandate the need to supply the registration number and a reason for that is so that reviewers can look at the original research protocol and determine if, and how, the researchers have deviated from their original research plan. This can have significant implications for understanding the validity of the results. It is quite common for authors to not disclose a secondary outcome.  Readers need to have secondary outcomes identified as such to understand whether the data set, as originally designed, was capable of accurately answering this new question the authors are now plugging simply because in looking for one thing they found something else that was (perhaps more) interesting. If you remove the clinical trial number it might render it impossible for the reviewer to do due diligence. However, if you provide the number, the identity of the study researchers is there for all to see.

These points are, of course, applicable to clinical medical journals. Journals in other subject domains will almost certainly have similar, but different, anonymizing issues. For instance, some fields, such as ecology, very actively support open data deposition. Reviewers following the trail to the open dataset will likely uncover the identity of the authors. Unfortunately, this is the inherent tension between openness and double anonymizing papers.