Why peer-review may be worth persisting with, despite everything
January 21, 2014
Regular readers will remember Jennifer Raff’s guest post on the PeerJ blog, How To Become Good At Peer-Review; and my response to it, Three points of disagreement. Today I read a very different take on this piece by Chorasimilarity, who is a frequent commenter here at SV-POW!: Two pieces of all too obvious propaganda.
Chorasimilarity starts by taking the original piece to task — fairly, I think — for its opening statement that “peer review is at the heart of the scientific method”. It’s true that the scientific method is something rather different. But as I argued in Science is enforced humility, peer-review is part of the scaffolding that prevents individual scientists from running away with their own ideas, unchecked by consensus wisdom.
Chorasimilarity then goes on to make a stronger criticism of peer-review:
Peer review is an idea based on authority, not on science [...] the quote mentions that “one’s research must survive the scrutiny of experts before it is presented to the larger scientific community as worthy of serious consideration”, which would be just sad, dinosaurish speaking, if it would come from an old person who did not understood that today there is, or there should be, free access to information.
As we’ve discussed here before, having been through peer review certainly does not mean we can trust a published paper. People do sometimes talk as though this is the case, and it’s an absolutely fallacy that we should be quick to rebut whenever we encounter it.
But it does have a weaker, yet still non-negligible, value.
The real value of peer-review not as a mark of correctness, but of seriousness. Back in the original SV-POW! series on peer-review (Where peer-review went wrong, Some more of peer-review’s greatest mistakes, What is this peer-review process anyway?, Well, that about wraps it up for peer-review), I likened peer-review to hazing:
The best analogy for our current system of pre-publication peer-review is that it’s a hazing ritual. It doesn’t exist because of any intrinsic value it has, and it certainly isn’t there for the benefit of the recipient. It’s basically a way to draw a line between In and Out. Something for the inductee to endure as a way of proving he’s made of the Right Stuff.
So: the principle value of peer-review is that it provides an opportunity for authors to demonstrate that they are prepared to undergo peer-review.
When I first wrote that, I wrote it in a spirit of cynicism and in frustration that so much of the effort that goes into the process is thrown away and that the results are so arbitrary. Those are real and serious complaints, but I’ve since come around to the idea that peer-review is useful in that the hazing aspect enables it to clear a much lower bar. Being prepared to undergo peer-review really is a mark of seriousness.
I would imagine that everyone involved in dinosaur research occasionally gets unsolicited emails from cranks and from as-yet unpublished amateurs. One of the most reliable ways to distinguish the two groups is this: serious amateurs are trying to figure out how to get their work into peer-review, while cranks are either actively avoiding it or not even aware of it. That’s why the web is full of sites like Dinosaur Home, with all its fine pictures of pebbles, which can continue on their merry way free of scrutiny.
I do think that the benefits of traditional peer-review are usually greatly overstated and the costs (both direct and indirect) underestimated. But I’m coming down on the side that its barrier-to-cranks effect might just tip the balance in favour of retaining it.
Think your work has scientific value? Good. Prove it, by showing it to professionals. If you won’t do that, then the rest of us don’t need to expend mental energy on taking you seriously.