Why peer-review may be worth persisting with, despite everything
January 21, 2014
Regular readers will remember Jennifer Raff’s guest post on the PeerJ blog, How To Become Good At Peer-Review; and my response to it, Three points of disagreement. Today I read a very different take on this piece by Chorasimilarity, who is a frequent commenter here at SV-POW!: Two pieces of all too obvious propaganda.
Chorasimilarity starts by taking the original piece to task — fairly, I think — for its opening statement that “peer review is at the heart of the scientific method”. It’s true that the scientific method is something rather different. But as I argued in Science is enforced humility, peer-review is part of the scaffolding that prevents individual scientists from running away with their own ideas, unchecked by consensus wisdom.
Chorasimilarity then goes on to make a stronger criticism of peer-review:
Peer review is an idea based on authority, not on science […] the quote mentions that “one’s research must survive the scrutiny of experts before it is presented to the larger scientific community as worthy of serious consideration”, which would be just sad, dinosaurish speaking, if it would come from an old person who did not understood that today there is, or there should be, free access to information.
As we’ve discussed here before, having been through peer review certainly does not mean we can trust a published paper. People do sometimes talk as though this is the case, and it’s an absolutely fallacy that we should be quick to rebut whenever we encounter it.
But it does have a weaker, yet still non-negligible, value.
The real value of peer-review is not as a mark of correctness, but of seriousness. Back in the original SV-POW! series on peer-review (Where peer-review went wrong, Some more of peer-review’s greatest mistakes, What is this peer-review process anyway?, Well, that about wraps it up for peer-review), I likened peer-review to hazing:
The best analogy for our current system of pre-publication peer-review is that it’s a hazing ritual. It doesn’t exist because of any intrinsic value it has, and it certainly isn’t there for the benefit of the recipient. It’s basically a way to draw a line between In and Out. Something for the inductee to endure as a way of proving he’s made of the Right Stuff.
So: the principal value of peer-review is that it provides an opportunity for authors to demonstrate that they are prepared to undergo peer-review.
When I first wrote that, I wrote it in a spirit of cynicism and in frustration that so much of the effort that goes into the process is thrown away and that the results are so arbitrary. Those are real and serious complaints, but I’ve since come around to the idea that peer-review is useful in that the hazing aspect enables it to clear a much lower bar. Being prepared to undergo peer-review really is a mark of seriousness.
I would imagine that everyone involved in dinosaur research occasionally gets unsolicited emails from cranks and from as-yet unpublished amateurs. One of the most reliable ways to distinguish the two groups is this: serious amateurs are trying to figure out how to get their work into peer-review, while cranks are either actively avoiding it or not even aware of it. That’s why the web is full of sites like Dinosaur Home, with all its fine pictures of pebbles, which can continue on their merry way free of scrutiny.
I do think that the benefits of traditional peer-review are usually greatly overstated and the costs (both direct and indirect) underestimated. But I’m coming down on the side that its barrier-to-cranks effect might just tip the balance in favour of retaining it.
Think your work has scientific value? Good. Prove it, by showing it to professionals. If you won’t do that, then the rest of us don’t need to expend mental energy on taking you seriously.
Sauropod Vertebra Picture of the Week 
January 21, 2014 at 9:12 am
Thank you Mike, I fully agree with your post. There is value in peer-review, even more if we leave aside the authority aspect. I say show the work to everybody, not only to professionals, put it somewhere where there are technical possibilities to comment on it, to review it or to take it and use it in other works. This is easy, technically one needs two things: the possibility to comment and a CC-BY license.
One can go even further: as happened with medieval commentaries on the ancient Greek philosophy, or the Bible, there is intrinsic value in the commentaries themselves. Peer-review can become a genre. With few exceptions, most of the work done for peer-reviewing goes to trash in this “hazing ritual” which leads to publication. By making the peer-review open, instead of using it as a bottleneck on the path to publication, this genre might flourish.
January 21, 2014 at 9:15 am
Needless to say, I strongly agree that throwing away all the work that goes into a review is stupid. It’s one reason that we did this and then this
January 21, 2014 at 12:30 pm
[…] post is motivated by the Mike Stay’s Why peer-review may be worth persisting with, despite everything and by comments at the post Two pieces of all too obvious […]
January 21, 2014 at 6:05 pm
As one of those supposed “authorities”, I can guarantee that I have had my share of manuscripts that were rejected. Some I have managed to salvage through revision, some remain asleep on my hard drive (I think I have five). I’ve endured the peer review process for over 40 years. Yes, I still get irritated with reviewers. I have had some reviews that were good and caught serious mistakes that I would not want out there published for ever. Consider how E.D. Cope could have saved himself embarrassment and lots of money had his Elasmosaurus manuscript been peer reviewed. The head on the wrong end would have been caught and not be something that still haunts him over a hundred years after his death. Do YOU want something similar?
I still rail against “small minded idiots who obviously didn’t understand the subject” :-) The worse reviewers are those who nitpick in order to give the impression that they did a thorough job reviewing (but of course, I am sure I have been guilty of that). One of the hardest part about reviewing is helping authors to whom English is a second language: trying to figure out what they are trying to say and help them say it, even if I don’t agree.
The hardest part about the review process, and why so many rail against it, is that it is a blow to your ego: you write what you consider a masterpiece, only to have someone (like me) trash it. This can be so emotionally crippling that some graduate students/professionals publish very few papers during their lives. As a result, I often (not always) cushion my comments to new writers that the comments are meant to be constructive, to make the manuscript better. I usually tell them to look at the comments, then ignore the manuscript for a month or more to put some emotional distance before trying to revise it. Then, it is possible to consider the review comments without seeing red. This is advice I still practice. Some comments clearly indicate the reviewer did not understand, so if they didn’t, it is possible others won’t either. Now is your chance to rewrite for clarity.
The all time worse manuscripts to review are those of authors trying to sound scientific. Don’t. Write simply. Don’t try to impress me with your writing, you WILL fail. For new writers (i.e., students), you can find more on this and other topics in my paper on “How to write a scientific paper” at http://www.mediafire.com/?01ytqituko7p0
January 21, 2014 at 6:14 pm
I’d agree with pretty much all of that, Ken. I suppose the only exception would be that there are some reviewers (Jerry Harris leaps to mind) who are very nit-picky, but do it with such a good spirit that it’s super-clear that the nit-picking is intended to improve the manuscript, not be a barrier to it.
November 6, 2014 at 8:06 am
[…] very helpful reviews that we received, and then also submitted for peer-review. We do still want that “we went through review” badge on our work (without believing it means more than it really does) and the archiving in PubMed Central and […]
February 4, 2021 at 4:41 pm
[…] real value of peer-review is not as a mark of correctness, but of seriousness” (from this 2014 post). If other people want me to part with my precious time to engage with their work, they can darn […]
March 9, 2021 at 2:06 pm
[…] the publish-first lifecycle could be exploited by cranks. If the willingness to undergo peer-review is the mark of seriousness in a researcher — and if non-serious researchers are unwilling to face that gauntlet — then a venue that lets […]