Filter-then-publish vs. publish-then-filter
May 10, 2012
How things have always been
Traditional scientific journals ask peer-reviewers to do two things: assess whether a manuscript is scientifically sound, and judge whether it’s sufficiently important to appear in the particular journal it’s been submitted to.
So I could have sent my 2009 paper on Brachiosaurus to Nature, and the reviewers would (presumably) have said “this is good science, but not exciting or sexy enough for Nature“. My article would have been filtered out of Nature, which after all is very limited for space. Instead, I sent it to the Journal of Vertebrate Paleontology, where the exciting-and-sexy bar is calibrated differently, and it passed both halves of the peer-review test.
Enter PLoS
The great insight of PLoS ONE was to recognise the two-pronged nature of peer-review, and to tease them apart by discarding the second prong completely. Its guidelines for reviewers are clear:
Unlike many journals which attempt to use the peer review process to determine whether or not an article reaches the level of ‘importance’ required by a given journal, PLoS ONE uses peer review to determine whether a paper is technically sound and worthy of inclusion in the published scientific record. Once the work is published in PLoS ONE, the broader community is then able to discuss and evaluate the significance of the article (through the number of citations it attracts; the downloads it achieves; the media and blog coverage it receives; and the post-publication Notes, Comments and Ratings that it receives on PLoS ONE etc).
I like to think of this as “kill ’em all and let God sort it out”, but a less colourful description that has caught on is “publish then filter“. This name contrasts nicely with the traditional model, which can be called “filter then publish“.
The models compared
On the whole, traditionalists prefer the older model, because when filtering is done in advance by professionals it saves them from having to do their own filtering.
Or does it?
No.
There was a time when it probably did: when to keep up with a field, it would be sufficient to read (or at least scan) the articles in a handful of the discipline’s top journals. But those days are long gone. I took a random selection of ten PDFs from my own library, and checked what journals they were in. In that sample, only a single journal came up more than once: there were two papers from Acta Paleontologica Polonica. The others were from Acta Geologica Sinica, the Anatomical Record, Animal Behaviour, Comparative Biochemistry and Physiology, Herpetological Conservation and Biology, the Journal of Vertebrate Paleontology and the Quarterly Journal of the Geological Society, plus a dissertation from the University of Flensburg. (And I am one of the most narrowly focussed researchers you could meet.)
In the face of such a flood of information, no-one can read everything that’s made it through the filters into all their favourite journals. So in practice what actually happens is that each of us filters again — finding relevant publications in a huge range of journals by the social web we’re in: mailing lists, blogs, Twitter, and so on. I believe some people even use FaceBook.
A tentative conclusion
So the real choice is between publish-then-filter or filter-then-publish-then-filter.
Put that way, I’m not sure I see very much value in that first filtering phase. I know it’s going to let through a ton of stuff that I don’t care about — all the palaeobotany papers in Palaeontologia Electronica, for example. But that pre-filter is also bound to stop a lot of stuff that I would care about if it were published. If JVP rejects someone’s unexciting paper on a partial Brachiosaurus specimen because it’s not sufficiently exciting, that may be good for the journal’s “prestige” (whatever that means) but it certainly doesn’t serve me as a researcher: I want all known specimens to be published.
So I am coming round to thinking that the PLoS way is best: if a paper is good science, then why even bother thinking about its likely impact? It’s not like that’s something we can expect to guess accurately, anyway. Just publish it and let the ashes fall where they may. The world will figure out for itself whether it’s worth reading and citing.
Sauropod Vertebra Picture of the Week 
May 10, 2012 at 8:34 pm
For society journals like JVP, the decision to reject an “unexciting paper on a partial Brachiosaurus specimen” is not just about ‘prestige’. Publishing a print journal costs money, and therefore there are limits to the number of pages the journal can publish a year. As a result, they can’t publish every scientifically sound paper that comes across the editorial desk, so some tough decisions have to be made. And these decisions relate to significance and breadth of audience for each manuscript, which is a sensible way to pick the papers you’re going to accept if you have limits on the number of pages you can publish.
Now, I know your answer is going to be: “That’s just another reason why PLoS One has an advantage, because they’re electronic-only, so they don’t have to worry about printing costs and limits on the number of papers published a year.” And you’re right – but that is a different discussion from the publish/filter question.
That being said, I generally am not a fan of framing any of these discussions as JVP vs. PLoS One, because I do think they serve different (albeit overlapping) purposes, and both have a place in our field. Furthermore, if one says, ‘PLoS One is superior so I’m only going to publish my descriptive manuscripts there’, I think there is the danger of supporting a journal monoculture. To me, plurality of journal choices for publishing is an extremely important thing to preserve.
May 10, 2012 at 8:47 pm
I’d agree with every single statement in that comment, Randy. I probably would have said most of it myself had I not been trying to keep this post reasonably short. Yes, physical space limitations change the game. And while I do see that as another reason to favour a PLoS ONE-like model, I wouldn’t want the result of that to be an all-PLoS ecosystem.
Down the line, I guess I would like to see JVP move to PLoS ONE-like model — online only (unless people want to pay for nicely printed copies), no limits on number of papers, length of number of figures, and making its money from publication fees and maybe sponsorships. Then again, you could say that I’ve just described Palaeontologia Electronica, and you’d be about right. (Heinrich, do you know if PE reviews on “impact” and “importance”, or only on science?)
May 10, 2012 at 8:51 pm
I think the counter to that argument is simple: why doesn’t JVP publish online as well as in print? The articles that are truly groundbreaking get into the print version, the ones that “do science” just as well get to be published online as long as they fit within the scope of the journal.
Obviously this would be more difficult for more generalist journals like Palaeontology, but for a specialist journal like JVP, it would make even more sense: if you (as a journal) reject a paper because it’s not “special” enough, the paper probably gets published someone else, and that other journal gets those citations. If, however, you publish that paper online, you get to keep good science in house and strengthen the brand.
May 10, 2012 at 8:53 pm
Matt BK, I can’t believe I never thought of that — it seems so obvious now!
Does anyone know of a journal that does this? Publishes many articles online, and only a selection of the sexiest in a printed edition?
May 11, 2012 at 12:32 am
>I think the counter to that argument is simple: why doesn’t JVP publish online as well as in print? The articles that are truly groundbreaking get into the print version, the ones that “do science” just as well get to be published online as long as they fit within the scope of the journal.
Heh, I just thought of this moments before I read it. But the follow-up argument is the clincher – the journal gets to keep the citations (not that journal-based metrics are all that great…)
I find it unfortunate that there is a natural instinct to equate ‘filter’ with ‘peer review’ (I confess to it myself when reading the headlines) which is especially common when mentioning e.g. the arXiv with its almost non-existence barrier to posting (usual caveats about endorsement model here). It would be good to come up with a way of describing the ‘publish then filter’ model in a way that makes it very obvious that peer review _has_ taken place in the sense that scientific accuracy has been checked. maybe a ‘verify, publish, then filter’ rather than a ‘verify-and-filter then publish’.
May 11, 2012 at 6:50 am
You’re right, David, that “filter” is an imprecise word in these “publish-then-filter” and “filter-then-publish” terms. Because both of the functions of peer-review that we mentioned result in filtering. To be really precise, we’d have to call the traditional model “filter-on-both-science-and-impact-then-publish”, and the PLoS ONE-like model “filter-on-science-then-publish-then-filter-on-impact”. But you have to admit that these are not snappy labels!
(And to be clear: peer-review does of course also have other functions besides filtering. Done well, it should result in a manuscript that is better than the one that went into the process.)
May 11, 2012 at 11:59 am
Another option, not dissimilar to what you are saying is to have an open access online journal where you can pay for a print copy by subscription. There are places where this makes a lot of sense – and it may work well if mixed in with membership in some places as well.
Or more sophisticated – have a premium print on demand service that just creates a print version that is optimised for you. Mike can have JVP without the plants for instance.
More generally I’m struggling with the concept of a journal that doesn’t actually have an electronic presence? Did I read that right?
May 11, 2012 at 12:09 pm
Yes, that is potentially an appealing idea. There are already no plants in JVP, of course, since its the journal of vertebrate palaeontology, but that way I could dump the fish as well.
… but as I say, it’s potentially an appealing idea. In practice, I don’t imagine I would ever actually order a custom issue. Having big, heavy hardcopy objects lying around the place seems such a 20th century thing to do. I guess I might order one issue a year, with only the dinosaur papers — or if there are too many to fit in a single issue, only the saurischians, or only the sauropods. But even then I struggle to think when I’d actually look at it in preference to the PDFs. I basically never look at the journal hardcopies I already own, which includes things like the Wilson and Sereno (1998) sauropod phylogeny monograph.
Finally:
I think you’re asking whether JVP has no online presence? No, that’s not the case! When David Roberts asked “why doesn’t JVP publish online as well as in print?” I assume he meant why doesn’t it publish additional papers online, beyond those that make it into the print edition.
May 11, 2012 at 4:03 pm
Just a quick note to say that I think most of us under-estimate how much the current filtering systems direct you to some content. According to Pubmed, from 2010 to 2011 there were 1799636 abstracts published or something like 4930 abstracts per day. Of these we pick a group of journals to pay attention to and hope that self-selection from the part of the authors and the editors criteria help to create spaces (journals) where we can find what we are interested in. The current filtering systems suck in many ways … but they are doing something.
May 11, 2012 at 5:25 pm
I wonder how right you are, Pedro. Personally I get all my article recommendations from peers in one way or another — mailing-lists, blogs, Twitter, direct personal “have you seen this?” messages — and I don’t use journal ToCs or alerts at all. But maybe the peers that I get my recommendations from are using the journal system. Seems more likely that they’re using subject alerts, though, rather than anything journal-based. (Or course most of them are getting their recommendations from their friends and colleagues, and so on, but the chain has to start somewhere.)
Think of it this way: PLoS ONE publishes 300 articles a week, so presumably no-one reads a weekly ToC email. Yet somehow we all become aware pretty quickly when a relevant PLoS ONE paper comes out. How does that happen?
May 14, 2012 at 8:51 pm
I question your faith in the ability of open access publishing to satisfactorily distribute the goods and services of research to its professional consumers. Don’t you think this Libertarian (read: Romantic) belief is a bit dated by now? After all, we’ve seen what happens when the gatekeepers are eliminated: the marketplace becomes chaotic and serves no one well. Why should ‘laissez-faire’ research publication work any better than ‘laissez-faire’ capitalism? Both assume there’s no need for gatekeepers and that value can only be determined through the marketplace. The recent SV-POW! flame war reminds me of a bunch of economists fighting over theory. Best to have at least some gatekeepers to slow down the pace of progress in useful ways. This helps control questionable squabbling by those making side bets at the margin.
Once again: Please stick to the subject of sauropod vertebra, a topic of endless fascination even when the phylogenetic going gets rough and the topic becomes remarkably subjective.
Love your blog of course or I wouldn’t have followed it so closely all these years. Maybe in your spare time you could write a tutorial for us to explain how cladistics differs from phenetics, which of these is more properly termed “evolutionary systematics,” why determination of taxons is difficult with cladistics and why Creationists welcomed cladistics because it declared that modern man has no ancestors preserved in the fossil record, and so on. All of this confuses me. I would also be interested in your estimation of how subjectivity enters into assessing all this (Life is messy after all), how species defined by one method should not be treated as if defined by another, how species can change names by branching or geographical isolation without showing morphological change and, generally, why numerical taxonomy can beat Garry Kasparov without introducing problems of its own, numerology being a messy business at best. Oh yes, and morphology’s role, SV-POW!’s main topic.
Cress Kearny
May 14, 2012 at 10:47 pm
I think you’re confused. This article is nothing to do with open access. You can have OA filter-first (e.g. PLoS Biology), OA publish-first (e.g. ArXiv), non-OA filter-first (e.g. JVP) or indeed non-OA publish-first (certain journals from New Mexico). Openness is entirely orthogonal to filtering precedence.
Again, you seem to be confusing two quite separate things. Whether we publish OA or barrier-based is one question; whether publishers are regulated or operate in a free market is another. It’s certainly true that open access tends to produce a freer market because it reduces lock-in; but however much you don’t like free markets you’re surely not arguing that monopolies are better? Because that’s what we have now. Once we have a functional market rather than the current disfunctional one, we can fruitfully debate how much and by whom it ought to be regulated.
Justify your assertion that the slowing-the-progress-of-science role of gatekeepers is a net positive. I don’t think you’re going to be able to.
Balls to that. This blog is about whatever Matt and I want to blog about. If you don’t want to read the non-sauropod posts, you have my permission to ignore them.
These would all be good topics, and we may well around to them in time. But it won’t be soon. We have more urgent fish to fry.
May 16, 2012 at 6:57 pm
The conclusion I come to is that there is far too much research going on.
May 16, 2012 at 9:47 pm
Er no.
One can never have too much research.
It keeps life interesting.
And when all the results are freely available to everyone life may get very interesting indeed.
LeeB.
May 16, 2012 at 10:02 pm
I was assuming (hoping) that Don was being cleverly facetious.