How to become good at peer-review: three points of disagreement
January 20, 2014
Jennifer Raff wrote a useful guest post on the PeerJ Blog: How To Become Good At Peer-Review. Most of its advice is excellent, and I’d heartily recommend it to anyone starting out on reviewing. But there are three points where I disagree with it. Here are the three things Jennifer said, and my counter-points.
1. Communicating with authors
“Don’t communicate with the authors about their manuscript. All thoughts and comments on it should only go to the editor.”
This may be different in different academic fields, but I’ve been contacted by reviewers of my material, and contacted the authors of papers I’m reviewing, too. Palaeo may be less formal in this respect than fields such as medical research. It’s often useful, for example, to get the authors to send higher resolution versions of the specimen photographs than the downscaled ones the journal passes on; or to get the manuscript in a read-write format that lets you more easily add notes and corrections. Most importantly, I’ve sometimes had to send my marked-up copy of the manuscript directly to the corresponding author because the journal’s automated system has no way to attach it to the formal response.
Perhaps the idea that you shouldn’t communicate with authors comes from confidentiality concerns. But I know who the authors are. (There are no palaeo journals that do double-blind reviewing, and it would be impossible any in a field small enough that you pretty much know who everyone is and what they work on.) And since I never review anonymously, I don’t mind them knowing who I am while I am still doing the review.
In the end, one of the main goals of peer-review — I would say the main goal — is to help the authors make their work the best it can be. Often contacting them directly is the more effective way to do that.
2. Novelty
“Ask yourself whether the questions the authors are addressing are really advancing the field in a meaningful way. This does not mean that an article has to be completely novel, but it does mean that the work contributes to the sum of knowledge in the field and does not, for example, simply repeat well known results.”
I only agree with this for certain values of “well known”. In experimental sciences, replication is hugely important, and it’s one of the worst consequences of the prestige-obsessed journal system that it’s so hard to get a replication published. You could almost say that an experimental result that’s only been published once is worthless.
Equally important, or maybe even more important than replication, is the failed replication. When Doyen et al. (2012) tried and failed to replicate the findings of Bargh et al. (1996) on psychological priming, it was an important check on the influence of an article that has been cited more than 2,500 times. Bargh himself was not happy about it, but to quote a much-loved SV-POW! maxim due to Tom Holtz, “Sorry if that makes some people feel bad, but I’m not in the ‘make people feel good’ business; I’m a scientist.”
So a reviewer should only complain about lack of novelty if the experiment has already been replicated several times. (There’s no value in a research paper showing that large and small cannonballs fall at the same speed from the top of the leaning tower of Pisa.)
3. Changing the subject
“Can you think of a better way to address the research questions than what the authors did?” … “You have every right to ask the authors to do a different experiment.”
Ugh. I just hate this. There is literally nothing I detest more in a review than “You should have written this different paper instead”. Please reviewers, review what’s in front of you, not what you would have done instead.
If you think of another approach that you think is promising, by all means suggest it as a followup project. But please in the name of all that we hold dear, don’t let it be a roadblock that delays this work from being published.
Sauropod Vertebra Picture of the Week 
January 20, 2014 at 12:34 pm
Re: point 3
The other issue, aside from the reviewer essentially moving the goalposts by doing this, is that quite often in smaller labs the person who did the majority of the research is no longer there once the paper is in a state to be submitted to a journal.
Staff turnover is high in the life sciences. You are expected to move on soon after doing your PhD, for example. Therefore, while it may sound the most reasonable request in the world in your head – often there is simply nobody left (or no spare capacity) to do the extra work you are blithely asking people to do. This leads to a missive delay on the submitters part whilst people either struggle to fit the experiments into their schedules, or the lab head can employ someone to do the work for them.
January 20, 2014 at 5:52 pm
Mike,
I don’t quite agree with you on point #3. If the paper sets out to test a certain question or hypothesis, and the authors conduct an inappropriate analysis/experiment/study to test that question/hypothesis, then the reviewer absolutely should request they do a different analysis/experiment, etc.
Similarly, I have run into the case where a paper as written does not ask a broad enough question for the journal its submitted to, but if they tweaked the focus of the paper or did a couple of additional analyses, then it would be appropriate for that journal. And I think its constructive to point that out.
January 20, 2014 at 10:12 pm
yep on 1, yep on 2, yep on 3!
220mya: note that you’re adding an assumption here! The original post did not suggest an “inappropriate” method.
I have had “write a different paper” reviews a few times. In each case, the reviewer was simply assuming that I have a few million Euros and several months to spend just so that I can attempt to find the World Formula and describe it in one go. Interestingly, whenever I pointed this out to the editors, they invariably agreed with me.
January 20, 2014 at 11:54 pm
[…] UPDATE (20.10.14): I can’t explain to myself why Mike Taylor does not detect this, behind the bland formulations. He does, however, makes good points here. […]
January 21, 2014 at 8:55 am
[…] guest post on the PeerJ blog, How To Become Good At Peer-Review; and my response to it, Three points of disagreement. Today I read a very different take on this piece by Chorasimilarity, who is a frequent commenter […]
January 21, 2014 at 7:13 pm
I mostly agree with you on #1, although my only hesitation is that it potentially opens the door for bad actors to bully authors with whom they disagree. I.e., Let’s say a grad student authors a taxonomic opinion that Reviewer 2 strongly disagrees with. Reviewer 2 (who was shown to be wrong by strong evidence in the manuscript) is in a huff that the editor didn’t adequately consider the objections, so they contact the student directly with a harshly worded email. This reviewer is a potential reviewer on grants and future papers and is a senior editor at another journal, etc., so the student isn’t in much of a position to call BS on this rogue reviewer. These kinds of power plays (and they do happen, albeit thankfully rarely) are part of why many consider it inappropriate for reviewers to contact authors directly.
On the flip side, I have also experienced very good communications with reviewers prior to hearing back from the journal handling my paper. This allows me to start addressing reviews quickly, speeding up the process, etc. This _almost_ makes me wonder if it wouldn’t be productive for journals to have the option to pass reviews along to authors as the reviews come in. . .maybe with a “do not communicate with the editor until a final decision is reached” policy for the authors to avoid unfair pressure to accept a paper.