Peer review for beginners

As a member of the first-ever group of editorial fellows at Annals of Internal Medicine, Yuri Matusov, MD, FACP, got an in-depth look at what makes an academic medical journal tick.

Over the 12-month program, from July 2023 to June 2024, he and three other participants learned peer review from the inside out, evaluating manuscripts being considered for publication in Annals as well as abstracts submitted to ACP's annual abstract competitions.

Inspired by the experience, and by their mentor, Annals Deputy Editor Vineet Chopra, MBBS, MD, MSc, FACP, the editorial fellows published an Ideas and Opinions piece in Annals in November 2024, using what they'd learned to offer advice to early career physicians and others who might be new to peer review. “We hope our perspective will encourage participation in peer review, reduce associated anxiety, and improve the quality of completed reviews,” they wrote.

Dr. Matusov, the article's corresponding author and an academic pulmonary and critical care physician and assistant professor of medicine at Cedars-Sinai in Los Angeles, recently spoke to I.M. Matters.

Q: How can early career physicians get started with peer review, and where do most peer review requests come from?

A: For some journals, you can sign up to be a peer reviewer. It doesn't mean you'll be selected, but it puts your name into a bank of people they can call on to do peer review. … If you have a mentor who's a more senior person, they often will get asked to do peer review, and it can be a very good learning experience and a good way to get into it to start doing peer review together. A lot of journals will allow that, as long as you tell them upfront. Not only will this get you exposure to doing peer review, but it will teach you how to do peer review correctly, assuming your mentor is a person who does peer review well. The last way is as you publish yourself, eventually you'll just start to get invitations, because a lot of journals will look for people who are published in the field of the article that they're soliciting a review for.

Q: What are some things to consider before accepting or declining a peer review invitation?

A: One thing to consider is that the article should be within your field of knowledge. When we start out as early career physicians, we don't have that many areas of expertise, and so you should try to peer review articles where you have a little bit more comfort, a little bit more understanding.

Second, you shouldn't review, or you shouldn't agree to review, if you don't have the time. There's one big exception to this rule: If it's for either a big-name journal, or a journal where you want to publish in the future, you should prioritize doing that review and try to figure out a way to make the time to do it, because in the end there's going to be a net benefit to your career. Even though the authors of the paper won't know who you are, the journal editors will.

In our paper, we encourage people to approach peer review from the standpoint of constructive criticism that makes the paper better, rather than finding ways to tear it apart. Doing that is very difficult, because essentially what you're doing is you are almost putting yourself in the place of the person who wrote the paper and you're saying, “OK, how can this paper be better?” Or, if you have significant issues with it, “How can this paper be saved?” That's a very time-consuming activity.

Third, you should make sure that you don't have any conflicts of interest with the paper, with the authors, or with the material in any way. Don't review if you have conflicts of interest.

Q: How can early career physicians determine whether they have a conflict of interest, and what should they do if they're not sure?

A: Conflicts of interest are somewhat nebulous things. There's some that are sort of obvious, where you have potential monetary gain from the impact of the paper, either positive or negative. That is not a paper that you should review, because you're going to be biased, even if you have the best of intentions.

There are some conflicts of interest that are a little bit more subtle. If you have a personal relationship with some of the authors, that can skew your review. That can be difficult, because if you are reviewing a paper for a very niche field, it's very likely that within a few years of starting in that field as junior faculty, you probably know a lot of people on a personal level. I don't necessarily think that that's an absolute reason not to review papers, but just keep in mind that your personal relationships shouldn't affect the way that you review the manuscript, again in a good way or a bad way.

Finally, if you have a very strong personal opinion on the topic, and the paper is arguing very strongly for or against, be careful reviewing that paper because you may find your opinion of the paper or your review being swayed by your own take on the material, independent of the scientific validity of the work of the authors.

If you feel like you may have a conflict of interest but you're unsure, you can tell the editors of the journal the situation, and often they can provide some guidance.

Q: Anonymity is usually a key part of peer review, but are there other areas of confidentiality that those new to the process might not be aware of?

A: Yes. You shouldn't discuss the data or the work with anyone outside of the editors of the journal. The manuscript represents the hard work and efforts of the authors, and they deserve the opportunity to have their work evaluated independently and fully vetted before it's put out into broader circulation. Avoid sharing details of the submitted manuscript with your colleagues who are not involved in the editorial process. If you print copies of it, avoid leaving them in places where they can be accessed by other people.

Also, avoid having direct communication with the authors. For the most part, when you're reviewing manuscripts, you're trying to be a third party that's independently assessing the work. If you have questions about the manuscript, address them through your comments to the authors through the journal portal itself. Don't send emails to the authors and ask them clarifying questions. That's not the way this is supposed to work.

Q: How can a new reviewer distinguish between which comments are appropriate for the authors and for the editors?

A: I've been guilty of this myself before I knew any better, but avoid putting the same comments in both the authors' and the editors' sections. That's just redundant. The comments to the authors should be based on the strengths and weaknesses of the work, what they can do to make the work stronger, and what you think are major issues with the work, for example how the manuscript or the data interpretation or various sections of the manuscript can be strengthened. Avoid saying to the authors whether or not you think the manuscript should be published. That's not your job or your responsibility.

In the comments to the editors, you can give a more frank opinion of whether you think that the manuscript merits publication. You can point out things that you think are major potential ethical issues or issues related to data, misinterpretation or plagiarism, or other very red-flag issues.

It's very important that you maintain professional decorum in communicating with both the authors and the editors. I've gotten many reviews on stuff that I've written where you can clearly tell that the person reviewing your work is just trying to tear it apart, and that accomplishes nothing. You have helped nobody by doing that. You've not made the work any stronger, you've just made the authors feel bad. Even if you think that the manuscript is terrible and that the work doesn't merit publication, people have still spent hours of their time putting this together, and you should be cognizant of that and show people respect.

Q: What are some of your favorite things about peer review?

A: It's made me a better consumer of medical literature, but I think it's also made me a better researcher physician in my own career. My peers doing amazing work is an incredible thing to see. It just goes to show me that this is doable, this is something that you can potentially accomplish. I think a lot of junior physicians, myself included, often have a sort of crisis of confidence in regard to whether or not what they're doing is valuable or whether the scientific work that they're trying to publish is meaningful. It can be very, very rewarding to see your peers creating great work, writing great manuscripts, presenting very interesting data.

Also, I do some medical education for residents and fellows in regard to interpretation of medical literature, and it's made me a better educator, because it's shown me the pitfalls that people sometimes fall into when they're designing their studies, or when they're writing up their work.

Peer review is often seen as sort of thankless work, and people, I think, are increasingly getting pulled to other commitments that are sometimes financially motivated and more career motivated. But doing high-quality peer review is hugely beneficial for an early career physician as well as for the authors and for the world of medical publication in general.

Since COVID in particular, and with the advent of AI, there's a lot of information out there that's not complete or not entirely accurate. I do think it's our job, as physicians who are involved in this process, to make sure that the stuff that's out there, the stuff that's being consumed by our colleagues and also by the lay public, is accurate and complete as much as possible.