What would happen if I publicly posted the reviews I receive?
December 23, 2012
After the authors’ own work, the biggest contribution to a published paper is the reviews provided, gratis, by peers. When peer-review works as it’s supposed to, they add significant value to the final paper. But the actual reviews are never seen by anyone except the authors and the handling editor.
This is bad for several reasons.
First, good reviewers don’t get the credit they deserve. That’s unfair on those who do a good job — who generously invest a lot of time and effort in others’ work.
Second, bad reviewers don’t get the blame they deserve. That leaves them free to act in bad faith: blocking papers by people they don’t like, or whose work is critical of their own; or just doing a completely inadequate job. Because there are no negative consequences for doing a bad job, people have no external incentive to straighten up and fly right.
Third, the effort that goes into reviewing is largely wasted. Often the reviews themselves are significant pieces of work (that’s certainly true when I’m the one giving the review) and the wider community could benefit from seeing them. Frequently reviews contain extended discussion, not only of the paper’s subject matter but of scientific philosophy such as approaches to taxonomy or narrative structure.
Fourth, editors’ decisions remain unexplained. Most editors handle manucripts efficiently and fairly, but there are cases when this isn’t the case — as for example when I was one of three reviewers who wholeheartedly recommended acceptance but the editor rejected the paper. Even discussing that situation was difficult, because the reviews in question were not available for the world to read.
Fifth, and more general than any of the above, the reviewing process is opaque to the world. In times past, logistical reasons such as lack of space in printed journals meant that the sausage-machine approach to the review process was the only feasible one: no-one wants to see what goes into the machine or what goes on inside, we only want the final product. But we live in an increasingly open world, and consensus is that pretty much all processes benefit from openness.
There are various initiatives under way to change the legacy system of reviewing, including F1000 Research and the eLife decision-letter system. But at the moment only a small minority of papers are submitted to such venues.
What to do about the others?
And so I found myself wondering … what would happen if I just unilaterally posted the reviews I receive? I already make pages on this site for each of my published papers (example): it would be easy to extend those pages by also adding:
- The submitted version of the manuscript
- All the reviews I received
- The editor’s decision letter
- My response letter to the editor
- The final published paper.
I know this is “not done”. My question is: why not? Is there an actual reason, other than inertia? Wouldn’t we all be better off if this was standard operating procedure?
[Note that this is orthogonal to reviewer anonymity. As it happens, I think that is also a bad thing, but it’s independent of what I’m proposing here. I could post an unsigned review as-is, without revealing who wrote it even if I knew.]
Sauropod Vertebra Picture of the Week 
December 23, 2012 at 11:27 am
I’ve often thought of doing this. As a reviewer, I wouldn’t have any objection – and I can’t see any reasonable objection anyone could have. I imagine editors might feel a little uncomfortable because their decisions are inevitably pretty subjective.
To be honest, I find the whole process frustrating and incredibly inefficient. As a reviewer, I care about what’s in the paper, not where it ends up being published. Instead, what often happens is you spend ages writing a review, the editor rejects the paper, the authors resubmit elsewhere without changing the paper, and it ends up being published a year later without your review having any impact other than delaying publication.
The system should just be that you submit a paper, the editor gets reviews and gives you an opportunity to address them, and the final version of the paper is published pretty much automatically, alongside the (signed) reviews, rebuttal, and the original version – at which point it would be open for post-publication review. It would mean everyone reviewing fewer papers but having a greater incentive to do a good job. It would also mean that if authors decide to ignore issues in the reviews, anyone reading the paper can see the reviews and make up their own minds.
December 23, 2012 at 11:29 am
I started posting my reviews this year:
http://proteinsandwavefunctions.blogspot.dk/search/label/reviews
The only thing that happened was a very courteous letter from PLoS ONE asking me very nicely to remove the name of the editor from the post. I don’t understand it since PLoS ONE lists the editor when the paper is published. But I did it anyway and will also do so for papers published in other journals.
I only published in PLoS ONE this year so I don’t know how or if other journals would react.
Since PLoS ONE has a comment section I have now posted links to the reviews there, so people can get the whole history of the paper directly from the journal site.
December 23, 2012 at 12:29 pm
Not a whole lot. I’ve done it here and there with little effect. Only once when I posted one quite obviously incompetent/lazy review (together with two ok reviews) of a grant proposal, the grant agency reacted by saying that they’d use my post to improve their peer-review process.
That’s about it for me, but I don’t do it regularly: with the number of revisions we often have, the versions, letters and reviews are usually much, much more to read than the actual paper!
December 24, 2012 at 2:44 am
One additional reason why it would be in a reviewers’ best interest to have reviews posted – fairly frequently, there are justifiable complaints that the editor and/or authors ignored major suggestions or objections. Posting reviews would add an extra level of transparency in this case (although some authors wouldn’t post if it showed just how tenuous their work was!).
As a reviewer, I might modify some aspects of my review if I knew it were to be posted. Occasionally I will throw in unpublished tidbits if I think they would be useful or informative to the authors, and I would be somewhat more hesitant to do that in some cases.
As a courtesy to the reviewers and editors, it might be a nice thing to add a statement in your cover letter and/or manuscript saying that you will post the reviews at your personal website, if the journal doesn’t have an option. Although I personally have no objection to my reviews being posted, because they are not usually prepared for public consumption it would be a little off-putting to see it just pop up on-line someday. Could there also be copyright issues involved in posting something without explicit consent?
December 24, 2012 at 5:48 am
There certainly would be copyright issues as the reviewer, in the absence of other contractual arrangements, would own copyright in their review of the paper. Whether or not the posting of a review would fall under “fair use” or not, I don’t know. It is something to consider. It would also be a good idea to check the journal’s submission requirements because they make confidentiality of things like the reviews and correspondence with editors a condition of submission. I have no idea if any journals do that, but it should be checked. Nobody needs legal hassles – except lawyers, of course. :-)
Transparency is almost always good for process and it would sometimes be fascinating to see how the process actually goes.
January 1, 2013 at 7:24 am
Not in the field, but following SV-POW with interest… keep the good work up!
Anyway, it occurs to me: don’t you owe the reviewers the secrecy? I mean, in the complex real world we actually live in, wouldn’t some reviewers be less frank if they knew their reviews will be made public? Even if it’s “anonymous”, I’m certain you can figure out in some cases who the reviewer is… and some reviewers, though certainly not all, may have a problem with that. In an ideal world they shouldn’t, but as I said, in the real world…
January 1, 2013 at 3:39 pm
Do we “owe reviewers secrecy”? I’m not sure. I think that, if they don’t elect to sign their reviews, then we owe them anonymity, but it doesn’t follow that the reviews themsleves should be kept secret.
You ask: “wouldn’t some reviewers be less frank if they knew their reviews will be made public?” Given what I have seen of “frank” reviews, such moderation would not be a bad thing. The negative but constructive reviews I’ve received are worthy of being shared because they are a credit to their writers. The mere hatchet-jobs ought to be published for the opposite reason.
January 7, 2013 at 1:19 am
I just found some discussion about posting reviews (from Science) here, esp in the comments:
http://phylogenomics.blogspot.com/2012/01/cracking-microbial-code-pam-ronald.html
Also kind of a warm fuzzy story about PLoS ONE.
January 10, 2013 at 2:34 pm
EMBO Journal has publishing reviewers comments, and the responses of the authors (in addition to other transparency measures) for at least 3 years. Both the reviews and the responses add value to the final paper, I find, and help to demystify the process for students etc. I don’t know why all society journals don’t do it. Journal of Neuroscience has “no confidential reviewer remarks” which is a good start.
http://www.nature.com/emboj/about/process.html
February 18, 2013 at 11:13 am
[…] among the most interesting things about PeerJ is its use of open peer review: reviewers are encouraged (though not required) to disclose their identity, and authors are […]