Some more of peer-review’s greatest mistakes
August 6, 2012
Last time I argued that traditional pre-publication peer-review isn’t necessarily worth the heavy burden it imposes. I guess no-one who’s been involved in the review process — as an author, editor or reviewer — will deny that it imposes significant costs, both in the time of all the participants, and in the delay in getting new work to press. Where I expected more pushback was in the claim that the benefits are not great.
Remember that the benefits usually claimed for peer-review are in two broad categories: that it improves the quality of what gets published, and that it filters out what’s not good enough to be published at all. I’ll save the second of these claims for next time. This time I want to look at the first.
The immediate catalyst is the two brief reviews with which a manuscript of mine (with Matt as co-author) was rejected two days ago from a mid-to-low ranked palaeo journal. I’m going to quote two sentences that rankle from one of the anonymous reviewers:
The manuscript reads as a long “story” instead of a scientific manuscript. Material and methods, results, and interpretation are unfortunately not clearly separated.
This, in the eyes of the reviewer, was one of the two deficiencies of the manuscript that led him or her to recommend rejection. (The other was that the manuscript is “too long”.) But as I’ve said repeatedly here and elsewhere, all good writing is story-telling. Scientific ideas, like all others, need to be presented as stories in order to sink into our brains. So what’s happened here is that, so far as I’m concerned, the reviewer has praised our manuscript; but he or she thinks its a criticism.
It’s not clear what we should do in response to such a review. To get our paper published in this journal, we could restructure the paper. We could draw every observation, comparison, discussion and illustration out of its current position in a single, flowing argument; and instead cram them them all, out of their natural order, into the “scientific” structure that the reviewer evidently prefers. But I’m not willing to do this, because our judgement is that this would reduce the value of the resulting paper — making it harder to follow the argument. And I hate to do work with negative net value.
In this case, it’s even worse: the initial draft of this paper was in a much more conventional “scientific” structure. In this form, we submitted it to a different journal, whence it was rejected for completely different reasons which I won’t go into. Before submitting the new version, we re-tooled the paper to tell its story in the order that makes most sense. So the reviewer who recommended that the new version be rejected was (albeit unknowingly) requiring us to revert an older and inferior version of the manuscript.
So this is a case where reviewers’ comments really don’t help at all. (To be fair to the reviewer in question, he or she did also say a lot of positive and encouraging things. But they are really just decoration on the big, fat REJECT.)
And sadly this kind of thing is not too unusual. Other reviews I’ve been sent have (A) demanded that we add a phylogenetic analysis even though it could not possibly tell us anything; (B) demanded that we remove a phylogenetic analysis; (C) rejected a paper due to consistently misunderstanding our anatomical nomenclature; (D) rejected my manuscript because they’d never heard of me (ad hominem review); and and my all-time most hated favourite (E) basically told me to write a completely different paper instead.
(E) in particular is a disaster and makes me just want to throw my hands up in the air. Written a paper that analyses a diversity database to draw conclusions about changing clade sizes through time? Too bad, the reviewer wants you to write a literature review on how you assembled the database instead! Written a paper showing that two species are not conspecific? Tough luck, the reviewer wants you to write a paper on the meaning of “genus” instead! I just loathe this. I am increasingly of the opinion that the best response may be “I’ve written that manuscript too, it’s in review elsewhere” or similar.
So those kinds of reviews are a complete waste of time and energy.
Where is the value, then? Well, I’ve mentioned Jerry Harris several times before as someone whose reviews are full of detailed, helpful comments that really do improve papers. I aspire to review like Jerry when it falls to me to provide this service, and I hope I can be as constructive as he is. But really, this is a tiny minority. Most reviews that avoid the traps I mentioned above don’t have much of substance to say. Some “reviews” I’ve received have been only a few sentences long: in such cases it’s hard to believe that the reviewers have actually read the manuscript, and very hard to accept that they have life-or-death power over its fate.
—
I’ve been wary of writing this post because of the fear that it reads like a catalogue of whining about how hard-done-by I am. That’s not the intention: I want to talk about widespread problems with peer-review, and I wanted to make them concrete by giving real examples. And of course the examples available to me are the reviews my own work has received. Let me note once again that I have made it through the peer-review gauntlet many times, and that I’m therefore criticising the system from within. I’m not a malcontent claiming there’s a conspiracy to keep my work out of journals. I am a small-time but real publishing author who’s sick of jumping through hoops.
Next time: what peer-review is really for.
Sauropod Vertebra Picture of the Week 
August 7, 2012 at 3:20 am
This looks to be a highly entertaining series.
“I want to talk about widespread problems with peer-review, and I wanted to make them concrete by giving real examples.”
Ironically I’ve been criticized for this very thing in rejections of my own paper (with Marjanovic)-
“Furthermore, the tone of the paper may be a turn-off to many readers. The authors may be trying to convince readers that [initials redacted] are careless and sloppy scientists. But the major point of the present paper is much more convincing if those scientists are not (i.e., instead, this is a common problem that everyone should be aware of, not just sloppy scientists). In short, the authors could make their paper much more compelling and readable by focusing on this useful methodological point, rather than focusing on the putative sloppiness of these other authors.”
August 7, 2012 at 6:36 am
I seems as if the only question remaining is, “Is it fixable?” Maybe that’s the third installment.
But evidence from real papers accepted or rejected (or, as seems more common, both) cannot carry the argument. For each paper actually written and submitted, many thousands that might have been were not. A few of those should have been submitted too, but the rest are finely crackled psychoceramics whose nature we can only guess at by looking at … well, the internet.
Every thoughtful observer agrees that the follies of democracy would be ultimately damning were not all the other alternatives even worse. We can evaluate one system only by detailed, evidence-based comparison to others, but each shred of evidence we have has cost untold rivers of blood. To begin, we must acknowledge that all viable alternatives are, at best, almost as bad. The conclusion turns on whether it seems slightly less bad than another, or even worse. There’s no limit on how much worse it may be.
August 7, 2012 at 8:22 am
Mickey says:
It’s a tricky line to walk. Leave out the examples and an issue seems too abstract to be convincing; include them and it risks looking like a personal complaint. I’m not sure what the answer is (as you can tell from this post!) but I suspect the best we can do may be just to be as careful as possible with tone — to make sure the paper spells out that the examples you give are just examples of a broader phenomenon.
(To be fair, one of the review comments I’ve had that was a real help was on one of my very first submissions, pointing out how unnecessarily critical the tone of the writing was. When I re-read the manuscript, I saw that it was true, and that the critical tone added nothing to my actual argument. It was easy to tweak this, and did result in a better paper.)
[BTW., Mickey, I redacted the individuals’ initials from the quote of your review, since their particular identities aren’t important here.]
August 7, 2012 at 8:30 am
I agree with pretty much all of that, Nathan. It does worry me that, while the current system does sort of, more or less, work, there are an infinity of alternatives and many of them are worse. It bothers me that there’s no way to determine by limited experiment which alternative will yield the best results. The only way to find out is by travelling into the future, at one second per second.
Still: my underlying point is that while five years ago I accepted that the current system is pretty close to optimal, I no longer feel that way.
As you guessed, I plan to look at alternatives in part 3.
August 7, 2012 at 9:28 am
Because peer review is probably not going to go away, I think the best we can do is to try and educate people about how to properly and carefully write reviews. I get the impression that many reviewers turn on asshole mode and end up saying many unnecessarily mean and totally unhelpful things. When I first started up I had numerous negative and nasty reviews from an individual (pre-submission); they were trying to help, but just lost their temper and wrote mean comments. I’ve also had reviews (positive and negative) that were literally a few non-committal sentences, without signatures – so I couldn’t even ask in the future to not use them due to their lameness. The best one was a recent review where the reviewer thought my research was uninteresting and inadequate because I failed to study a number of unpublished (!!!) fossils from the other side of the planet which nobody in my hemisphere.
Perhaps a great post topic would be “how to write a constructive review”. I think, for one, that it is an absolute necessity to include detailed comments (point by point, line by line – corrections, rewording, comments, etc.). The author can’t read your mind if you chose not to include details in your review, and as someone who’s received such constructive and detailed reviews, it really does help. The other big thing is tone – which you brought up above; many authors don’t realize (myself included) that word choice can be easily interpreted as nasty (So and so (2009) neglected… failed to… etc.); there are perhaps certain cases where it’s admissable or justified, but on the whole this should probably be avoided (we’re a small field, and we want to stay friends… right?). There are more constructive ways of pointing out @#$%-ups than using such language. If you want to, great, but many in the field can view it as juvenile or immature. This goes for reviewing as well – I think a well-crafted review should take quite a bit of time to do properly.
I think of giving a review as an opportunity: I usually end up having to read some additional material and learn quite a bit. If you carefully review something, and sign it so the author knows you’re the one responsible, it’s also an opportunity to come across as very competent and perhaps surprise someone (on the flipside, if you sign a shoddily executed review, there will be a couple people out there who may think you’re a complete idiot and/or asshole). Lastly… as a reviewer, that means somebody (editor or author) actually cares what you think, which is a pretty satisfying feeling.
August 7, 2012 at 9:38 am
Thanks, Robert. I agree that “how to write a review” would be a useful topic; but before we can talk about that meaningfully we need to figure what reviews are actually for, and that will be the subject of the next post.
August 7, 2012 at 10:23 am
Mike, you keep talking about criticizing the ‘system’ – in fact you appear to be criticizing particular reviewers (and praising others). To me, this doesn’t seem a very convincing indictment of the system of pre-publication peer review.
August 7, 2012 at 3:49 pm
Richard Van Noorden, I can only assure you that it is indeed the system as a whole that I’m criticising. I am trying to make that criticism concrete by citing examples, and to make it fair by citing counter-examples. I’m sorry if the effect of that is to make the whole thing feel like a parade of anecdotes.
Richard Butler, thanks for the inside line on the saltasaurine pneumaticity paper’s journey through review. You are absolutely right that in principle “perceived “impact” and importance, which is a separate issue to whether or not peer review is a good idea”; but in practice of course the two are strongly conflated. That’s a big part of the problem, and will be the subject of part 3 of this series.
August 7, 2012 at 8:25 pm
“[BTW., Mickey, I redacted the individuals’ initials from the quote of your review, since their particular identities aren’t important here.]”
Eh, I’ve criticized two of the three papers publically anyway (the other is Marjanovic’s forte).
“It’s a tricky line to walk. Leave out the examples and an issue seems too abstract to be convincing; include them and it risks looking like a personal complaint. I’m not sure what the answer is (as you can tell from this post!) but I suspect the best we can do may be just to be as careful as possible with tone — to make sure the paper spells out that the examples you give are just examples of a broader phenomenon.”
I think in general, scientists need to take criticism of their work less personally. I’ve had friendly interactions with one of the redacted individuals and admire parts of their character, but they still neglected major responsibilities when making their data matrix. This manuscript is regarding my recent crusade against matrices that leave huge swaths of well known codings uncoded, if you haven’t guessed. Which is itself a great example of the failure of peer review, with such uncoded matrices having appeared in Science and Nature.
In this particular case, I think calling individual papers out is important for two reasons. One, coding only half of the codable characters for a taxon in your analysis is not simply a mistake, nor is it excusable when the characters are obvious and non-controversial (e.g. tibiofemoral ratios) and apparent in references you used to make your own character list. Thus it was a conscious choice to not do the expected work. If you don’t call such authors out, they’re just going to do the same thing again (which they did). Two, other authors will not realize which matrices are useless due to not including half the data they should, and will use those matrices to add their own taxa to (which has happened).
That was somewhat of an aside, but while most of the reviewers’ comments were useless, we did use one to improve the paper significantly, since I had the chance to do that time-consuming portion while Marjanovic tediously reformatted it for resubmission to another journal. In the ultimate irony though, despite our conclusion urging peer reviewers to check a few taxa for coding completeness/accuracy, neither reviewer noticed we had accidentally left out the first three lines of our first matrix, so it started with character 46 instead of 1. *sigh*
August 7, 2012 at 8:43 pm
Ugh. It sends a shiver up my spine every time I read something like this.
August 7, 2012 at 11:48 pm
Mickey, I kind of would like to back up Mike here when it comes to developing your tone. We had this discussion when remarking about Bhullar et al. on cranial morphometrics implying paedomorphosis in Avialae (with the conversation split offline and here).
When one acts in an aggressive tone, we tend to put our hackles up and act as though a thing of ours is being assaulted (or even we ourselves). We become defensive. I agree with Mike that it’s a tightrope act, but I should note that you and I have had this conversation elsewhere before; you seem not to agree any moderation of tone is required. Professionalism sometimes requires courtesy even to one’s ideological opponents, such that when two competing groups meet, they do not simply ignore one another or impair objectivity in the scientific front. As an example, Mike Taylor et al. versus Kent Stevens et al. for the positive approach of rival arguments; Aetogate resolving from the negative. I would rather pick the side that pretends there is NO conflict and would seek NONE anyways. Is it also possible you are not aware of your aggressive tone, or how overt it can be?
August 8, 2012 at 12:17 am
Out of curiosity, why didn’t you submit it to PLoS One?
August 8, 2012 at 12:41 am
Probably there’s a drug already in circulation that the publication should supply along with the manuscript, to take before proceeding with a review. Dispassionolol?
August 8, 2012 at 8:22 am
jacek asks:
I do sometimes wonder about that. There’s no one reason, and I admit it does seem like the most obvious venue. It may sound feeble, but one reason is PLoS’s use of the lame numbered-references system. The manuscript in question is a big one: it has 150 references and I guess about 500 citations to them. Not only am I seriously unenthusiastic about the massive amount of tedious clerical work involved in changing the manuscript to this form, I would be astonished if making the post-review revisions to a manuscript in this form didn’t result in introducing referencing errors.
“But, Mike, you should have used a reference manager like EndNote or Zotero! Then you could change the referencing style with the flick of a switch!”
Yes. That would have been a good decision to make back in 2008 when I started writing this thing.
August 8, 2012 at 8:22 am
With all that said, there’s a pretty good chance that it will, in the end, go to PLoS ONE.
Part of the reason we didn’t sent it to PLoS ONE last time (and I know this is going to sound terribly arrogant) is that PLoS ONE gets a lot of excellent papers now. We wanted to give a different journal, which we felt needed it more, the opportunity to publish our work. Of course it turned out that they didn’t even want our stinkin’ work.
August 8, 2012 at 4:13 pm
And now you know why my first paper will ge to PLoS ONE. I will probably try to submit elsewhere with one of my papers for the entertaining experience of what it will be like to try to navigate the maze of “meaningfulness” or such fiddlefaddle, but in the end I wish to support a journal whose aim is reader-chosen and cite-chosen impact, and whose effect is that Science, data collection, presentation is done without fear.
August 8, 2012 at 4:19 pm
Amen!
August 9, 2012 at 11:45 pm
In response to Jaime, I’m aware of my scathing tone in regard to ideas, but I remain courteous to the people themselves. Respect the person, but disrespect their ideas if deserved. Just look at any comment exchange I’ve had with e.g. Pickering. I’d say I’m more polite to him than many are, and certainly moreso than he’s been to me at times. In Bhullar’s case, he didn’t take my criticism of his work personally, we both cooperated to resolve our differences, and now have good relations with each other. I can’t see how it could have worked out better. If you want to argue Aetogate resolved poorly due to negative tone, I’m sure the SV-POWers would be happy to give their opinion.
August 10, 2012 at 9:08 am
No need for us to say more that subject, of course — I already said all I have to say about how that debacle resolved, or rather how it did not get resolved.
(That is the reason I am no longer an SVP member.)
August 10, 2012 at 12:50 pm
[…] Zivkovic (@BoraZ) August 7, 2012 New on SV-POW: Some More of Peer-Review’s Greatest Mistakes svpow.com/2012/08/06/som… See pt1: Where Peer-Review Went Wrong […]
August 12, 2012 at 3:05 am
[…] peer-review went wrong and Some more of peer-review s greatest mistakes and What is this peer-review process anyway? by Mike […]
August 19, 2012 at 12:57 pm
Accept or reject is an editorial decision. A reviewer can recommend a publication decision, but cannot make the decision. A reviewer’s role is to present a review. Period. Then it’s up to an editor to take the reviews together with their own evaluation and make a publication decision. These distinct roles should not be confused, but often are, by both reviewers and authors.
August 21, 2012 at 10:23 pm
Rich, I think you may be overstating the independence of the two. I would love to believe that editors perform independent evaluations, but the truth of the process is that editors depend heavily on reviewers to determine what makes the cut.
The responsibility is shared between reviewers and editors, and part of the reviewer’s responsibility is acknowledging that reviews are the sentry at the gates. Without a sound review, editors may never even consider the paper.
August 30, 2012 at 7:11 am
[…] effectiveness (or rather, ineffectiveness) of peer review and the standard publishing system (here, here and here are the most recent entries), and this included a small discussion on the form and […]
September 5, 2012 at 2:21 pm
[…] it has not seen any peer review, and as much as some of my colleagues hate peer review (here, here and here), it does sort out a lot of papers with major flaws. And it shows a lot of signs of being […]
September 5, 2012 at 3:58 pm
…WTF. Try different journals, if you can; they might select different reviewers.
Hey, it’s only been rejected once so far! :-) And that was mostly for not being breathtaking enough for the journal (no, it wasn’t Nature or Science or PNAS).
Our conclusion is that scary sloppiness is scarily widespread. That’s something we want people to know.
Add more examples to spread the perceived hostility, and sooner or later the paper is too long to be published in anything with an impact factor – but we want as many people as possible to know about this important methodologic issue.
Heh. Same for me, almost certainly with a different redacted individual!
Oh yes. I gleefully told this to the editor, saying he should feel free to pass it on to the reviewers. That was the first and possibly last time I replied to a rejection. =8-)
It’s not a good way of putting it to say we accidentally left it out; the table was repeatedly destroyed in sending it back and forth between our two MS Word versions – the last time this happened before submission, I evidently didn’t notice, and maybe I couldn’t even have.
BTW, reformatting it wasn’t all that tedious. I just had trouble bringing myself to do it, what with things to do in meatspace and… the glaring incompleteness of the instructions to authors for that particular journal, four years after they were changed the last time. I had to study at least two recent papers to figure out how to format certain things. *facepalm*
“We” might not even notice that “we”‘re being passive-aggressive here, Jaime.
Would you like to see our manuscript, if Mickey agrees, before we submit? Just so you know what you’re talking about, I mean? I’d trust you with it.
I still am so I can work from the inside. :-/
Their own evaluation? Except at the most specialized journals, most manuscripts are pretty far outside their specialties, so they reply pretty much exclusively on the reviews.
If one review says “revise” and the other says “reject with extreme prejudice”, the manuscript will probably be rejected, even if the second review is irrational; that’s because the editor so often can’t tell.
September 21, 2012 at 7:47 am
[…] mon post il y a quelques temps, et à une série de posts intéressants de Mike Taylor (posts 1, 2, 3), je me suis demandé à quel point les différences de point de vue sur l’expertise par […]
September 25, 2012 at 9:55 am
[…] time: some of the details of why my paper was rejected, and why I think they’re dumb reasons. In part three: what peer-review should actually be […]
September 26, 2012 at 7:13 am
[…] we went to visit the collections of the Yale Peabody Museum. And it’s the version that was cursorily rejected from mid-to-low ranked palaeo journal because a reviewer said “The manuscript reads as a long “story” instead of a scientific […]
October 3, 2012 at 10:51 pm
[…] my increasing disillusionment with the traditional pre-publication peer-review process [post 1, post 2, post 3]. By coincidence, it was in between writing the second and third in that series of posts […]
November 26, 2012 at 12:31 pm
[…] part 1, part 2 and part 3 from a few months […]
April 10, 2013 at 5:46 pm
[…] the narrative into a fixed set of sections (Introduction, Methods, Results, […]
January 21, 2014 at 8:55 am
[…] of seriousness. Back in the original SV-POW! series on peer-review (Where peer-review went wrong, Some more of peer-review’s greatest mistakes, What is this peer-review process anyway?, Well, that about wraps it up for peer-review), I […]
July 19, 2020 at 10:25 pm
This is a good read. I am undergoing the peer review process – finding people publish in better journal than me with inferior data and story compared to my work. It is frustrating.
I strongly encourage that all peer-review correspondence is also available for download with every paper. As we strife for openness in science, that is the only way forward to ensure some honest review process.
September 11, 2020 at 11:20 am
[…] is, I’ve done this myself. For example, back in 2012, I wrote about having a paper rejected from “a mid-to-low ranked palaeo journal” for what I considered […]