What are we going to call PLOS ONE-style peer-review?
April 24, 2015
When a paper goes for peer-review at PLOS ONE, the reviewers are told not to make any judgement about how important or sexy or “impacty” the paper is — to judge it only on methodical soundness. All papers that are judged sound are to be published without making guesses about which will and won’t improve the journal’s reputation through being influential down the line. (Such guesses are hopelessly inaccurate anyway.)
When PLOS ONE was new, this approach drew scorn from established publishers, but now those publishers all have their own journals that use similar editorial criteria (Nature’s Scientific Reports, AAAS‘s Science Advances, Elsevier’s first attempt, Elsevier’s second attempt, the Royal Society’s Royal Society Open Science). Those editorial criteria have proved their worth.
But what are we going to call this style of peer-review?
It’s not a new problem. I discussed it with with David Crotty three years ago without reaching any very satisfactory conclusion. But three years have not really helped us much as we try to agree on a term for this increasingly important and prevalent model.
What are the options on the table?
PLOS ONE-style peer-review. It’s a cumbersome term, and it privileges PLOS ONE when that is now far from the only journal to use this approach to peer-review (and may not even have been first).
Peer-review Lite. A snide term coined by people who wanted PLOS ONE to fail. It’s not a good description, and it carries baggage.
Scientific peer-review. This one came up in the discussion with David Crotty, but it’s not really acceptable because it would leave us still needing a term for what the Open Library of Humanities does.
Objective peer-review. This is the term that was used at the Royal Society meeting at the start of this week — the idea being that you review objectively for the quality of the research, but don’t make a subjective judgement of its importance. Several people didn’t like this on the grounds that even the “objective” half is inevitably subjective.
Any others that I missed?
I don’t have a good solution to propose to this problem; but I think it’s getting more urgent that we do solve it. We have to have a simple, unambiguous, universally understood term to understand a model of peer-review that is becoming increasingly pervasive and may well end up as the dominant form of peer-review.
Plough in — comments are open!
Update, 6pm
Liz Wager asked a very similar question four years ago, over on the BMJ blog: what to call the journals that use this approach to peer-review. Terms that she mentions include:
- “bias to publish” (from BioMed Central)
- “non-selective” (her own coinage, which she doesn’t like)
- “bumboat” (I can’t explain this one, you’ll have to read the article)
- “author-driver” or “author-focused” publication (AFP for short)
- “search-located” (which she coins, the dismisses as tautologous)
- “unconventional” or “non-traditional” (discarded as disparaging)
- “non-discriminatory”, “impartial” or “unprejudiced”
- “general” (dismissed as a non-starter)
- “broad-spectrum” (inapplicable to specialised journals)
And then in the comments various people proposed:
- “below the fold” journals
- “omnivorous” (I quite like that one)
- “alternative”
- “Voldermortian journals”, which I don’t understand at all.
- “Unfiltered”, contrasted with “filtered”
- “inclusive”, contrasted with “exclusive” (I quite like this, too)
- “high volume low hassle”
But there’s no conclusion or preferred term.
Sauropod Vertebra Picture of the Week 
April 24, 2015 at 1:16 pm
How about calling it “peer review” and then try to find a way to call the other ridiculous process we are so used to.
April 24, 2015 at 1:22 pm
It’s just “peer review”
The fluffy “is it sexy? is it impacty? is is appropriate for this journal?” is Editorial Review.
April 24, 2015 at 1:58 pm
What about something like ‘threshold review’? The idea is that a work that passes through this sort of review has at least met the minimum standards of rigor and accuracy to be judged worthy of becoming part of the scholarly conversation. Whether it has impact beyond that would be left to ‘post-publication peer review’.
April 24, 2015 at 2:11 pm
Exactly!
As Pandelis said
“How about calling it “peer review” and then try to find a way to call the other ridiculous process we are so used to.”
April 24, 2015 at 2:13 pm
PROPER peer review.
April 24, 2015 at 2:43 pm
A name for PLOS One style journals has emerged: megajournals. It isn’t ideal given that some of them aren’t large, but it seems to be sticking (see ‘A survey of authors publishing in four megajournals’ in PeerJ https://peerj.com/articles/365/ for example).
Given this, I suggest the term ‘megajournal-style peer review’. It is, after all, the defining feature of megajournals, so this term is specific and accurate. It doesn’t focus on any of the specific features of this kind of review, but that might be a good thing given that things like ‘objective peer review’ are subjective.
April 24, 2015 at 2:51 pm
Hi Mike–as noted in our earlier discussion, the real semantic problem is that you have an established practice in place, and a new one that (while still of high rigor) reviews for fewer criteria. This sets you up so that almost all of the proposed terms end up coming off as pejorative because they imply “lesser” in quantity, which can be easily, if inaccurately, interpreted as “lesser in quality”. It’s a tricky minefield to navigate as one writes or talks about peer review–you want to express the idea clearly without being insulting to anyone or belittling the deliberate choices the new process has made.
I’ve yet to find a good, short and catchy term that works, and usually end up with a longer and still somewhat awkward “peer review for accuracy but not significance” or some such.
How about giving credit where credit is due, and following in the lines of Xerox and Kleenex and doing as you suggest above, just calling it PLOS ONE Style Review? At this point most folks seem to know what you’re talking about when you say that. Might not make those behind all the other journals that have come along since then very happy though…
April 24, 2015 at 2:57 pm
Scientific, non-selective, peer review?
April 24, 2015 at 3:01 pm
I see, humanities. Scholarly non-selective peer review then?
April 24, 2015 at 3:12 pm
For me, the term “technical review” covers it. I make the distinction between gate-keeping review of the “is this cool enough for this journal” type, vs technical review that asks “is this sound science” – e.g. here http://blogs.bmj.com/bmj/2014/12/23/theodora-bloom-peer-review-and-gatekeeping/
April 24, 2015 at 3:18 pm
I personally like “peer review”, because there is no single style of peer review that defines every journal. [note: I am a volunteer editor for PLOS ONE] When it comes to reviewing for scientific soundness, rather than perceived impact, I agree with those above who consider these editorial criteria rather than a character of the peer review. Quite frankly, there have always been journals that focused more on perceived soundness than perceived impact (some regional science academy or institutional journals come to mind as examples).
So the short version…just call it peer review. Everything else is editorial.
April 24, 2015 at 3:18 pm
In the BMC Series, which pioneered this sort of review before PLOS One started doing it, we call it “inclusive”. See Di Marshall’s blog on the matter where she explains “Inclusive: The BMC series remains true to its origins and what it was launched to achieve, a complete and unbiased scientific record overseen by editors who do not make interest level a criterion for publication.” http://blogs.biomedcentral.com/bmcseriesblog/2014/05/14/what-is-the-bmc-series/
April 24, 2015 at 3:20 pm
I have used “Impact Neutral Peer Review” and “Objective Peer Review” before, but it is definitely the case that there is no accepted term in the community (at least not one which a large proportion of actual academics would recognize). Whenever I talk about it I always have to go on and clarify what it does / doesn’t do.
April 24, 2015 at 3:25 pm
It doesn’t really matter what you call it anyway. It’s clear now that it doesn’t work very well, and it’s on its way out. Post-publication review will replace it eventually.
April 24, 2015 at 4:17 pm
Yes, and formal journal “publication” will be replaced by “making public” to any digital medium (preferably free and handled by non-profit scholarly organisations) that ensures permanency of content, assigns DOIs, etc. Coupled with good quality post-publication review, who needs the journals anymore? :)
April 24, 2015 at 4:33 pm
Another suggestion: peer review for soundness, as opposed to peer review for significance. ‘Soundness’ captures most of what is meant by checking that the results justify the conclusions, regardless of how interesting the conclusions are.
April 24, 2015 at 6:22 pm
I’ve always referred to this as “Methodological Review”, which ensures that the methods used in the science are all sound. It’s more about whether the science was conducted properly and whether the arguments are sound. But I’d also throw a vote behind the other commenters wanting instead to rebrand the other form of peer review as “Editorial Review”. But you’re going to have a much harder time redefining the established definition of peer review than you are positively branding the new version.
April 24, 2015 at 7:38 pm
After a bit of thought I’ve come around to David’s suggestion – there’s a good chance that something like “plos-style peer review” (give or take capitalisation) is what we’ll end up with. We’ve had most of a decade to come up with a snappy name, and a widely accepted (or even locally accepted) one hasn’t emerged, which suggests it’ll be hard for anything to gain traction now…
I like “inclusive” but it doesn’t seem to quite capture the nuance of what we’re distinguishing. Something like “lightweight” is faintly perjorative; as noted, it implies a reduced threshold rather than passing one of two tests. “Limited” and “restricted” have the same overtones (though I’m fond of “restricted”).
As to “megajournal”, FWIW, I’ve always interpreted that as referring to the journal’s volume and range (many papers on a wide range of topics), rather than whether they adopt the new-style peer review. They generally *have* done so, but I think you could potentially run a smaller more selective megajournal and still justify that name. However, whatever form of review is used, it would seem strange to have a small journal referred to this way.
(huh, there’s another possibility – “new” review!)
April 24, 2015 at 7:52 pm
I promised to add my feedback here after an interesting twitter dialogue this morning (quite early in CA!).
My definition was “traditional pre-publication peer review without regard to the likely effect on journal impact factor”. Which isn’t very catchy!
Anna is right however, that although ONE’s peer review methodology feels somewhat traditional to me now (from a ScienceOpen perspective at least) it is a very long way from the original and still widely used version.
Perhaps what would be better is a definition something like “an updated version of pre-publication peer-review without regard to journal impact factor”. Still not very catchy.
However, I do really like “peer review for soundness not significance” which is a riff on Anna’s version above.
I then raised that while PLOS ONE does a fine job all round, it is more expensive than many newer market entrants, slower and the review process is not open.
Now change is afoot @PLOS “in the not too distant future” per this blog from CEO Elizabeth Marincola http://blogs.royalsociety.org/in-verba/2015/03/27/fssc-peer-review-with-elizabeth-marincola/.
My ask of Andrew Farke, as a ONE Board Member, was to find out a bit more about the direction and possible timing. I think that newcomers such as F1000R, PeerJ, The Winnower and ScienceOpen, all peer review reformers, will all get a nice boost when a key publisher innovates in this space too.
Hopefully that’s a reasonable summary of what’s on my mind.
April 24, 2015 at 7:53 pm
That said – having done a bit of poking around, it seems I’ve had an idiosyncratic reading of ‘megajournal’ all this time and the general consensus is in line with Anna’s reading. So I will happily defer on that one…
April 24, 2015 at 8:18 pm
How about, “sadly ineffective”. http://blogs.plos.org/mindthebrain/2013/10/01/why-i-accepted-a-plos-one-article-about-homeopathy-for-depression/
April 24, 2015 at 9:56 pm
How about “soundness review”?
April 24, 2015 at 10:14 pm
Wow, a lot of really interesting ideas here (and a couple of bad ones :-))
Keep ’em coming: I’ll try to summarise in a subsequent post.
My interim summary is: no consensus is emerging, at least not yet.
April 24, 2015 at 10:22 pm
BJ Nicholls, I don’t at all understand why you think the homeopathy paper indicate ineffective peer-review. By the handling editor’s account, it was a perfectly cromulent clinical trial, pre-registered, with transparent methods and no axe to grind; and it found (as we would expect) no evidence that homeopathic remedies are better than placebo. On what possible grounds should it have been denied publication?
April 24, 2015 at 10:44 pm
James Coyne has justified his acceptance of that homeopathy paper at http://blogs.plos.org/mindthebrain/2013/10/01/why-i-accepted-a-plos-one-article-about-homeopathy-for-depression/
Despite his admirable frank defence, I think the paper should not have been accepted, at least in the form that it appeared, for the reasons I give at http://blogs.plos.org/mindthebrain/2013/10/01/why-i-accepted-a-plos-one-article-about-homeopathy-for-depression/#comment-573416 and, in slightly more detail, in the comments at Plos One: http://www.plosone.org/annotation/listThread.action?root=86134
April 25, 2015 at 1:10 pm
The term “sound research review” aims to capture this focus on the validity of the research rather than its novelty and interest. http://blog.rubriq.com/2014/12/05/rubriq-adds-sound-research-stamps/
April 26, 2015 at 11:39 am
I would use “peer review” to refer to PLOS-style peer review, and “impact review” for the Science/Nature-style additional review. So a paper submitted to Science/Nature/etc would undergo both peer review and impact review in my terminology.
May 2, 2015 at 11:29 pm
Seems to me PLOS ONE’s approach is just “peer review”.
You need another term to describe what journals that try to foresee future impact and to enhance their own reputations are doing. Maybe “clairvoyant, narcissistic peer review’.
May 29, 2015 at 7:01 pm
[…] am using the term “megajournal” here to mean “journal that practices PLOS ONE-style peer-review for correctness only, ignoring guesses at possible impact”. It’s not a great term for […]
April 25, 2020 at 9:37 pm
[…] This is particularly odd when you consider the prices of the obvious alternative megajournals: […]