Reviewer Training as a Form of Engagement: A Summary of Training Programs

In this second of three posts on the theme of reviewer training as a form of engagement to both increase reviewer invitation acceptance rates and elevate review standards, we look at a variety of training programs that have already been implemented. This endeavor was not a methodologically driven academic exercise. Instead, the purpose of this post is simply to collate some potentially inspiring programs that journals and societies may consider emulating. The final post in this series will reflect upon the issues associated with implementing a peer reviewer-training program.

As many of the training programs listed in this blog post note, willing and able peer reviewers are increasingly hard to find. As similarly posited in the initial post on training as a form of reviewer engagement, there is a sense that if you offer a tangible educational benefit, especially one that involves direct interaction with trainees, such as a mentorship program, there is a possibility you can build loyalty while simultaneously inculcating the values of effective peer review.  After reviewing multiple peer reviewer training programs, it is fair to say this concept is not especially radical or original. For instance, while seeking a course provider for their peer reviewer training course, the European Geosciences Union explicitly linked training to boosting the number of qualified reviewers.

As journals express frustration regarding the difficulties of securing reviewers and as publishers and industry service providers scramble to find better, algorithmically driven, ways to identify reviewers, it is fair to say many journals do not do a good job of connecting and engaging with the reviewers they already have. An annual shout out is always good, and should be done, but probably only marginally fosters goodwill or boosts a sense of collaboration and brand loyalty between a journal and its constituents.

Naturally, there is a tendency for the best-resourced journals to take the lead in developing training initiatives. It is also likely, in a good number of cases, that they are the journals least in need of remedial efforts. Smaller or newer journals will be confronted with either a lack of resources to develop training or no obvious community to call upon and mold into a pool of engaged potential reviewers, especially if those journals are not backed by a supporting society. Nevertheless, that does not mean such journals should exclude themselves from training efforts. As some of the examples presented in this post illustrate, there are multiple ways to train reviewers. Many present similarly themed training materials that can be adapted and redeployed in a variety of journal contexts. Some organizations, such as publishers and industry service providers offer training that could be linked to or co-opted into some sort of supporting arrangement. Yet more examples see societies support their journals by providing training programs as a member benefit, rather than leaving the journal to fend for itself. Furthermore, there is nothing stopping journals from banding together and jointly offering resources or training programs. In short, the wheel does not need to be invented journal by journal.

One problem in devising any manner of training program, however, is that little- to no- research has been undertaken to determine collective reviewer core competencies, making this systematic review preprint by Willis et al. a rare source of information. Furthermore, there is no definitive evidence that peer review training works, though, of course, there is a vast, general, corpus of literature on the effectiveness of training, especially with regards to mentorship, beyond the narrow confines of peer review training. That reviewers need training at all has still never been scientifically confirmed though, anecdotally, most journal editors would agree that just because a researcher is a subject expert, it does not make them a skilled peer reviewer. Indeed, though researchers often learn by doing and reflecting upon what they learned, what is really needed is a structured training program that provides feedback and, to some degree, modeling of good practice. It is also likely impossible to train a general subject matter expert, such as a clinician, to be a strong methodological reviewer as well. This may suggest any training program will work best if it trains reviewers to be skilled in some tasks and self-aware in others to know they are at the limits of their knowledge base and to raise the flag for specialist reviewers, such as statisticians or patient reviewers to take over with a closer look. In a similar theme, Jigisha Patel’s opinion piece on providing specific training to review Randomized Controlled Trials is spot on with its assertion that specialized training, backed with skills appraisals and revalidation, could elevate standards. One could also contend that such specializing in training could posit a sense of clear purposefulness for reviewers, who would understand clearly both the task in front of them and why they were being asked in the first place.

Ultimately, as this is a blog post, our intent is to provide a short cut to several peer reviewer training resources for editorial offices, journals and societies that are looking to develop their own programs or refine what they are already doing. Likely, there are probably many more journals that are undertaking semi-structured or informal training behind closed doors. Some programs focus on highly technical matters or journal-specific needs. Others, plainly, are generic and may provide useful foundations upon which to build more specific resources. To aid comprehension, the resources have been grouped into several different categories. Membership of the category is not exclusive, and the program offered may well be the result of collaborative efforts from several different actors.

What we now attempt to do is review some of the most well-known, readily accessible and discoverable reviewer training programs broken down by the type of program provider. Within each section, the programs are roughly listed in order of comprehensiveness, with the top of the list prioritizing programs that provide general information, evidence-based training, a tangible reward, multiple methods of training, mentee recruitment through invited reviewer recommendation and participation as a mentor; and the list ends with programs for physician residents, and other student- / trainer-led programs.

Examples of Journal-Specific Training

The following list contains training programs that are typically resource driven – such as lecture slides, reviewer guidelines and reading materials. Some mentor-driven programs appear to have evolved from the unofficial practice whereby assigned reviewers pass on the job of completing a review to a student, often without credit, into a more formalized program of mentor-mentee instruction while completing peer review on a manuscript undergoing review.

A detailed tabular summary of these training resources has been placed here. You will be able to download the table from that link.

• BMJ Reviewer Training Resources – one of the most comprehensive sets of resources, originally developed for part of a randomized controlled trial and now being offered for individuals to pursue or for other publications to reference. –  https://www.bmj.com/about-bmj/resources-reviewers/training-materials
• Nature Masterclasses – subscription-based course only available to institutions. Individuals encouraged to refer the course to their institutions – https://masterclasses.nature.com/focus-on-peer-review-online-course/16605550
• Analysis of Verbal Behavior – mentorship program whereby junior scholars were paired with mentor to provide feedback and modelling. – https://doi.org/10.1007/s40616-022-00172-x
• Medical Care – mentor driven peer review with extensive array of online resources https://www.themedicalcareblog.com/peer-review-mentoring/
• Journal of Cardiac Failure – mentorship program accompanied by didactic online sessions; trainees work on live manuscripts  – https://hfsa.org/jcf-reviewer-mentoring-program
• American Journal of Pharmaceutical Education – mentorship program following completion of researcher/reviewer training programs provided by Elsevier (the publisher of the journal) and the Web of Science. Recognition of graduation from the training program – https://www.ajpe.org/reviewer-mentorship-program
• Clinical Journal of the American Society of Nephrology – early career mentorship program that has collected resources on peer review and recorded training podcasts –  https://journals.lww.com/CJASN/Pages/traineepeerreview.aspx
• Trials – declining reviewers can instead offer to mentor an early-career researcher to complete the review of a manuscript. –  https://trialsjournal.biomedcentral.com/about/peer-review-mentoring
• Advances in Nursing Science – mentorship program that acknowledges that to increase diversity and inclusion, the next generation of researchers must have a pathway towards being trained and recognized as reviewers; invited reviewers can offer to mentor a junior colleague to complete a review, thus intent is to model good practice – https://ansjournalblog.com/2019/01/02/ans-peer-review-mentoring-program/
• Earth System Science Data – Sharing of manuscripts with instructors who then teach their research groups. These instructors may be reviewers and the manuscripts may be those currently under review at ESSD.  Journal stresses its open data as a helpful resource to encourage the complete validation of a study  – https://essd.copernicus.org/preprints/essd-2022-387/essd-2022-387-manuscript-version3.pdf
• Gastrointestinal Endoscopy – selective mentorship program for junior researchers. Mentors are drawn from the highest performing reviewers. https://www.giejournal.org/article/S0016-5107(21)01724-7/fulltext
• Genetics – Online course and mentorship, selective program following applications – https://genetics-gsa.org/career-development/genetics-peer-review-training-program/
• Clinical Microbiology and Infection – discusses the need to train researchers to conduct peer review, promotes mentorship as a superior method of training in comparison to self-directed learning, and introduces the journal’s model of mentorship in which reviewers who decline an invitation have the option to recommend a mentee as an alternate reviewer – https://www.clinicalmicrobiologyandinfection.com/article/S1198-743X(22)00367-6/fulltext
• Clinical Practice in Pediatric Psychology – formalizing the process of a reviewer passing off a submission to a junior researcher to review – expectation is that the reviewer will review the peer review guideline with the junior reviewer first –  https://www.apa.org/pubs/journals/cpp/reviewer-mentoring-program
• JACC Case Reports – lectures and discussions accompanied by a mentorship program with practice on manuscripts currently under peer review – https://www.jacc.org/doi/10.1016/j.jaccas.2021.06.020
• International Journal of Radiation Oncology, Biology, Physics – Training program for residents; work on current submissions with feedback delivered on reviews completed; if 8 reviews completed in 2 years, participants will receive a certificate and their program directors will be notified – https://www.redjournal.org/content/review
• Neurology – typical mentor-mentee program for early career researchers from the leading journal in the field of neurology – https://blogs.neurology.org/rf/improving-peer-reviewing-through-paired-mentor-mentee-reviews-with-the-rfs/
• Journal of Arthroplasty – Mentorship by the best reviewers as identified by the journal – https://www.aahks.org/focal/joa-reviewer-mentorship-program/
• American Journal of Psychiatry Residents’ Journal – launched a journal whereby the early career scientists can undertake experiential learning by actually doing the peer review job in a live setting with guidance, the participants actively forming part of the editorial board – https://ajp.psychiatryonline.org/doi/10.1176/appi.ajp.2015.15101267
• Journal of Immunotherapy of Cancer (JITC) – training program for early career researchers including those with no peer reviewing experience with guidance provided one-on-one or in small groups – https://www.sitcancer.org/research/jitc-2/become-a-reviewer
• Wilderness & Environmental Medicine Reviewer-in-Training (RIT) Program – Currently promoted via word of mouth; interested RITs are instructed to email the journal’s managing editor to be registered. RITs are invited as the third reviewer (journal standard is to have 2 reviewers). RITs are identified as such when reviews are sent to the author, and they may receive additional feedback about their reviews from the editor-in-chief and possibly from an associate or section editor. WEM is currently revamping the RIT program and when complete will post on its website.

Examples of Society-Generated Training

During our search we found some instances where the society, rather than a specific journal, appeared to offer training. The level at which this training was pitched ranged from a clear member benefit to a general offering to the entire field of study, be they members of the society or not. In some instances (e.g. the American Speech Language Hearing Association) it seems the society is providing the content to serve its portfolio of journals.

A detailed tabular summary of these training resources has been placed here. You will be able to download the table from that link.

• American Chemical Society Reviewer Lab – one of the most prominent reviewer training programs, modular training resources with self-assessment tests to conclude – https://institute.acs.org/courses/acs-reviewer-lab.html
• American Aging Association – GeroScience Peer Review Training Program for early career, work on live manuscripts with a mentor – https://www.americanagingassociation.org/geroscience-peer-review-training-program
• American Society of Clinical Oncology – selective mentorship program with feedback on manuscript reviews; employed across all ASCO journals – https://ascopubs.org/reviewers
• American College of Clinical Pharmacy – selective mentorship program with online resources combined with mentor-feedback for two peer reviews completed – https://www.accp.com/store/product.aspx?pc=PRT22
• International Parkinson and Movement Disorders Society – mentorship program with extensive online resources – https://www.movementdisorders.org/MDS/Peer-Reviewing-Program.htm
• American College of Rheumatology – selective mentorship program for their portfolio of journals with additional online instructional materials to be read before mentorship starts – https://rheumatology.org/acr-journals-peer-review-mentoring-program
• American Speech Language Hearing Association – online, self-paced learning modules –  https://academy.pubs.asha.org/prep-the-asha-journals-peer-review-excellence-program/
• American Society of Agronomy, Crop Science Society of America, Soil Science Society of America journals – after initial completion of free Web of Science modules, begin review of submissions with mentors providing feedback on the review generated – https://www.agronomy.org/publications/journals/peer-review-mentorship/
• British Psychological Society – mentorship program with training videos and webinars provided by the Wiley Reviewer Academy – https://www.bps.org.uk/psychologist/journal-reviewer-mentoring
• Society for Neuroscience – mentor-led program with certificate awarded upon completion – https://www.jneurosci.org/rmp
• American Society of Therapeutic and Radiation Oncology – two training programs offered for two different journals. See above for the International Journal of Radiation Oncology, Biology, Physics. For Practical Radiation Oncology, typical mentor driven program with feedback given for completed reviews – https://www.astro.org/News-and-Publications/Journals/News/Reviewers

Publisher-based Training

Most of the major publishers offer training in some form or another. Much of it focuses on the core principles of good peer review that stretch across every field of study (be it art history or computational physics). The courses listed below appear to be free of charge unless otherwise stated:

• Elsevier Certified Peer Reviewer Course – certification program that provides general introduction to peer review; course developed by academic editors and Elsevier staff, over 4 hours of learning modules with a self-assessment module at the end. Unrestricted access to the learning modules. https://researcheracademy.elsevier.com/navigating-peer-review/certified-peer-reviewer-course
• Wiley – simple suite of training videos as part of a broader compendium of materials on reviewing articles; does not seem to be a discrete course in the manner delivered by Elsevier – https://authorservices.wiley.com/Reviewers/journal-reviewers/becoming-a-reviewer.html/peer-review-training.html
• Wolters Kluwer (developed in association with Editage) – basic course with freely accessible training modules and an advanced course with extra topics covered, such as methods and statistics, self-assessment modules. Advanced course costs US$150 – https://wkauthorservices.editage.com/peer-reviewer-training-course/
• Taylor and Francis – in-person or virtual workshops in English and Chinese, participants with the relevant skills and experience are then matched to journals (presumably Taylor and Francis journals); 6 online training modules are also offered –  https://editorresources.taylorandfrancis.com/reviewer-guidelines/peer-review-training/
• Institute of Physics – peer reviewer certification following training delivered through a variety of media including examples of good and bad reviews – https://ioppublishing.org/researchers/peer-review-excellence-online-course/
• Sage Publishing – a series of three current, free peer reviewer training webinars, among other resources.  Part 1 of the three part webinar was completed in May 2023 and a recording is available.  Reviewers can register for parts 2 and 3 which are scheduled for October 18, 2023 and November 8, 2023.  https://us.sagepub.com/en-us/nam/how-to-be-a-peer-reviewer-webinar“
• Springer Nature Resources for Peer Reviewers – links to both Springer and BMC resources on how to evaluate a manuscript and a third link to the Nature Masterclasses. These Masterclasses were referenced under journal-based training above – https://www.springernature.com/gp/reviewers?gclid=CjwKCAjwg-GjBhBnEiwAMUvNWzTxmJHNjN9BeQT-Xu5QZQdGFrbjxoly4IJQMmfGKq2tBWU8BTPm6RoCU08QAvD_BwE
• BMC Peer review – tips for junior reviewers – https://www.biomedcentral.com/collections/peerreviewtips
• BMC Blog series “How to Peer Review” – not a course per se, but a useful series of bits of information –  http://blogs.biomedcentral.com/bmcblog/tag/how-to-peer-review/
• SPIE/Researcher.Life – course is led by R Upskill experts which is part of researcher.life which, in turn, is part of Cactus (who also owns Editage), offers same basic and advanced courses as Wolters Kluwer, through the advanced course, which appears to contain similar course materials – costs US$100 – https://researcher.life/partner/spie

Funder-led Programs

• Canadian Institute of Health Research – participation by application in a mentor-driven program supplemented with online resources, recent Canadian federal grant recipients are eligible, each CIHR Project committee can host 2 mentees at a time; program is designed with Project grant review in mind, though with clear crossover appeal for general journal reviewer training – https://cihr-irsc.gc.ca/e/52323.html

Other Examples

• Clarivate’s Web of Science Academy (incorporating the former Publons Academy – multiple learning modules with open enrollment, certification of completion, Publons Academy is often cited and linked to by other older training programs; modules on scholarly peer review, how to be a peer reviewer mentor, research integrity – https://webofscienceacademy.clarivate.com/learn
• University of Alberta – libraries could be a source of training with both on-site and virtual training – https://ualberta.libcal.com/event/3697965
• University of Oxford, Mathematical, Physical and Life Sciences Division – university departments also likely offer in-person training, some more formalized than others if they are didactically driven rather than mentor-based; the Oxford course was selective and in-person – https://www.mpls.ox.ac.uk/training/mpls-training/our-courses/mpls-research-courses/how-to-peer-review-journal-papers
• Federation of societies special working group – EMEUNET (part of European Alliance of Associations for Rheumatology (EULAR)) – cross journal mentor-led program running since 2012; mentees work independently on a review under the guidance of a mentor. A research paper assessing the success of the program was published with helpful advice on how other journals could implement a training program – https://emeunet.eular.org/peer_review_mentoring_programme.cfm
• EQUATOR Network – clearinghouse of useful resources training programs may wish to incorporate or link to in their training resources – https://www.equator-network.org/toolkits/peer-reviewing-research/peer-review-training-and-resources/
• Cochrane – free, online course for biomedical reviewers provided by a special interest group of Cochrane; special focus on ophthalmology due to funding received from the National Eye Institute; course has been endorsed by 4 external-to-Cochrane journals; useful example of how a subject domain can collaborate to create training and elevate collective practice –  https://eyes.cochrane.org/free-online-course-journal-peer-review;
• COPE – static document on all ethical aspects of peer review from what ethical issues to look for through to the ethics associated with conducting a review –  https://publicationethics.org/files/cope-ethical-guidelines-peer-reviewers-v2_0.pdf
• BMJ Books “How to Survive Peer Review” – classic, now freely available, book on how peer review is conducted that can be read from the perspective of an editor, author or reviewer –  https://www.bmj.com/sites/default/files/attachments/resources/2011/07/wager.pdf
• CHAMP checklist for the assessment of medical papers – useful guide/resource for authors and reviewers; 30-item checklist related to the design and conduct, data analysis, reporting and presentation, and interpretation of a research paper for breaking down biomedical statistics issues – https://pubmed.ncbi.nlm.nih.gov/33514559/
• Sense About Science “Peer Review: the nuts and bolts” – summary, downloadable guide breaking down all aspects of peer review – https://senseaboutscience.org/activities/peer-review-the-nuts-and-bolts-2/
• PREreview (now supported by eLife) – interactive peer review mentoring program; “Mentees will be guided by their mentors in writing and posting preprint reviews that will be published with a digital object identifier”; strong emphasis on promoting DEI issues as part of the art of good peer review, rather than just focusing on methodology – https://elifesciences.org/for-the-press/d04c6551/elife-supports-prereview-s-new-peer-review-mentoring-program-for-early-career-researchers

Common Characteristics of Reviewer Training Programs

Some of the programs, such as those developed by The BMJ, are designed to appeal to anyone interested in improving their peer review skills. Overwhelmingly, however, the programs originate with early career researchers in mind, often under a mentor-mentee, experiential learning model, sometimes including an evaluation of what editors/course directors have determined to be a good review. For example, The BMJ also provides examples of editor feedback on peer review comments and a sample manuscript for any interested individual to review. The courses may conclude with a self-assessment test.

The need for early career training is obvious, especially with no standardized training, as many researchers may be disadvantaged by not having access to learning opportunities. As the American Psychological Association notes, training at graduate or post-graduate level is not always provided. Some courses seek to formalize the already existing informal practice of reviewers passing off the job of reviewing to their students by suggesting the originally assigned reviewers offer some sort of mentorship while completing the review, with the student then receiving recognition for completing the review. Many of these programs also stress that participating in a course offers the opportunity to open up new career advancement possibilities.

While some journal programs do not mention a reward, several programs – such as the one devised by Elsevier – offer participants a certificate or letter of completion, whereas others offer recognition in the journal, integration into a journal reviewer pool or various combinations of these rewards.  Uniquely, the American Journal of Pharmaceutical Education conducts a formal graduation from their reviewer mentorship program, and the Journal of Immunotherapy of Cancer offers recognition at its society’s annual meeting as well as addition to the preferred reviewer database.  Some societies such as the American College of Clinical Pharmacy and the American Society of Therapeutic and Radiation Oncology offer continuing education credits, and the latter also offers the opportunity to become a fellow of the society.  In addition to a certificate of completion, the American Chemical Society offers training in English and in Chinese and offers reviewer trainees who pass the final assessment an identifying badge in their submission system.  As mentioned in the previous post, journals can choose to collaborate to provide reviewer training to researchers.  The American Society of Agronomy, Crop Science Society of America, and Soil Science Society of America journals have done just that by uniting to provide mentorship to early career researchers, taking advantage of the Web of Science Academy curriculum as a basis for their program.  In return, participants receive credit on their Web of Science profile, in addition to a certificate of completion and enrollment as a peer reviewer by the appropriate journal.

What is less clear is whether any of the courses were evidence-based in their design and, crucially, who trains the trainers. Some programs such as the ACS Reviewer Lab and the course offered by The BMJ seem to have been constructed upon published research that has studied the impact of peer review. Most programs, alternatively, seem to be led by subject thought leaders and/or known high-quality reviewers. While mentorship and experiential learning under the guidance of a mentor can be successful, there is the potential for it to accentuate bias or prolong entrenched hierarchical views (“listen to me/trust me, I am an expert”). The art of peer review, of course, is particularly susceptible to this as there are no agreed-upon core competencies, no definitive training resources and little research into what constitutes a good review. A pertinent question to ask, therefore, is who trains the trainers? The Web of Science Academy was one of the few resources that appeared to address the mentor rather than just the mentee with a suite of learning modules.  Willis et al recently wrote an article about a risk of bias tool that they developed to assess the extent to which reviewer training programs are evidence-based.  The tool assesses factors such as whether more than one stakeholder group was involved in developing the reviewer training program, data was gathered for training development, pilot testing was done, learning objectives were presented, and whether learning was evaluated, as well as the method of feedback used in evaluation. In lieu of/in the absence of a training program for trainers, such a tool may be helpful for journals seeking a training program for their reviewers in the interim.

Conclusion

Training courses are somewhat commonplace and range in scale, knowledge delivery mechanism and intended audience. Many aim to combine didactic, self-paced, learning modules with mentorship. But, as this review of the courses shows, there are no uniform standards and no validation of the accuracy or appropriateness of what is taught. Nevertheless, many of these efforts offer possibilities to many that otherwise face no opportunity to learn, which in turn may impact on their overall success as a researcher by missing out on learning about key analytical skills. Interestingly, most of the programs tie training to both boosting the pool of available reviewers and raising collective skill levels, all concepts discussed in the first part of this blog series.

Nevertheless, many journals will struggle to develop their own resources, though setting up a network of mentors may be a more obtainable goal. The potential exists for collaboration. The most successful approach to collaboration may be at a cross-journal or subject matter level. Any higher level than that, such as publisher-led, though useful, may lack the specificity that is needed for domains of study because of their need to serve a general population. Perhaps the future lies in defining reviewer core competencies at a general level, establishing a code of ethics and then adding layers of subject specificity on top of those competencies. Journals can then add a further layer on top of that if they have very specific questions they wish their reviewers to address. Collaboration, of course, can collectively elevate a field while also allowing smaller journals to participate in training programs.Willis et al recently wrote an article about a risk of bias tool that they developed to assess the extent to which reviewer training programs are evidence-based.  The tool assesses factors such as whether more than one stakeholder group was involved in developing the reviewer training program, data was gathered for training development, pilot testing was done, learning objectives were presented, and whether learning was evaluated, as well as the method of feedback used in evaluation. In lieu of/in the absence of a training program for trainers, such a tool may be helpful for journals seeking a training program for their reviewers in the interim.

Acknowledgements:
Thanks to Stephanie Kinnan, Senior Managing Editor of Publications, American Society for Gastrointestinal Endoscopy for her thoughts on the Gastrointestinal Endoscopy peer review training program.

Origin Editorial is now part of KnowledgeWorks Global Ltd., the industry leader in editorial, production, online hosting, and transformative services for every stage of the content lifecycle. We are your source for society services, market analysis, intelligent automation, digital delivery, and more. Email us at info@kwglobal.com.