What is this peer-review process anyway?
August 7, 2012
Let me begin with a digression. (Hey, we may as well start as we mean to go on.)
Citations in scientific writing are used for two very different reasons, but because the two cases have the same form we often confuse them. We may cite a work as an authority, to lend its weight to our own assertion, as in “Diplodocus carnegii seems to have had fifteen cervical vertebrae (Hatcher 1901)”; or we may cite a work to give it credit for an observation or concept, as in “… using the extant phylogenetic bracket (Witmer 1995)”.
The conflation of these two very different modes of citation causes some difficulty because, while many authors would never cite (say) a blog post as an authority, most would feel some obligation to cite it in order to give credit. You might not want to cite this SV-POW! post as authority for a length of 49 m for Amphicoelias fragillimus; but you would hardly use the sacrum illustration from this post in your own work without crediting the source.
So the fact citations do two rather different jobs causes confusion.
When it comes to peer-review, things are worse: not only it trying to do two different things at once (filtering manuscripts, and improving the ones it retains) but the filtering itself has two components — deciding (A) whether a manuscript is good science, and (B) whether it’s “a good fit” for the journal.
What does “a good fit” mean? Anything or nothing, unfortunately. In the case of Science ‘n’ Nature, of course, it means “is about a new feathered theropod”. For Paleobiology, it seems to mean “has lots of graphs and no specimen photos”. In the case of other journals, it’s much less predictable, and can often, it seems, come down to the whim of the individual reviewer. In many cases, infuriatingly, it can be a matter of whether the reviewer’s guess is that a work is “important” enough for the journal — something that you can never tell in advance, but which only becomes apparent in the years following publication.
As a result, a perfectly good piece of work — one which passes the peer-review process’s “is it good science?” filter with flying colours — can still get rejected, and indeed can be bounced around from journal to journal until the author loses interest, leaves the field or dies; or, of course, until the paper is accepted somewhere.
(The already disheartening process of shopping a paper around multiple journals until it finds a home is made utterly soul-crushing by the completely different formats that the different journals require their submissions in. But that’s a completely different rant.)
This is the Gordian knot that PLoS ONE set out to cut by simply declaring that all scientifically satisfactory work is a good fit. Reviewers for PLoS ONE are explicitly told not to make judgements about the probable impact of papers, and only judge whether the science is good. In this way, it’s left to the rest of world to evaluate the importance of the work — just as in fact it always does anyway (through citation, blog discussion, formal responses, and so on).
But eliminating the “good fit” criterion still leaves PLoS ONE reviewers with two jobs: judging whether a manuscript is scientifically sound, and helping the author to improve it. I find myself wondering whether there might be a way to decouple these functions, too.
Perhaps not: after all, they are somewhat intertwingled. The question “is this scientifically sound” does not always receive a yes-or-no answer. The answer might be “yes, provided that the author’s conclusions are corroborated by a phylogenetic analysis”, for example.
I don’t have any good answers to propose here. (Not yet, anyway.) At this stage, I am just trying to think clearly about the problem, not to come up with solutions. But I do think we can see what’s going on with more clarity if our minds can separate out all the different roles that reviewers play. To my mind, dumping “impact assessment” from the review process is PLoS ONE’s greatest achievement. If we can pick things apart yet further, there may well be even greater gains to be had.
I guess we should be asking ourselves this: what, when it comes right down to it, is peer-review for? Back in the day, the filtering aspect was crucial because paper printing and distribution meant that there was a strict limit on how many papers could be published. That’s not true any more, so that aspect of peer-review is no longer relevant. But what else has the Internet changed? What can we streamline?
Sauropod Vertebra Picture of the Week 
August 7, 2012 at 11:35 pm
Still reading through this. I’d like to drop in a comment, though:
Citations should really be doing three things:
1. Shorten context to a reference, so that one need not cover broad ground when mentioning something (such as with your example of the EPB).
2. Give credit for ideas or concepts when the context is short enough to include in the work (this time, using the example of something simple like putting trunks on diplodocid sauropods).
3. Use for historical reference. Sometimes, a reference is there solely because you want the reader to follow those refs from the beginning and read the process through. Useful for subjects far more complicated than even something like the EPB, or for which books exist (why, if for any reason, would you cite a novel?).
But that’s what you should be using them for. There is also:
4. Padding your references. You can get carried away: not every reference in required, and when the same data you are referring to is covered in multiple papers, which largely cover the same ground in discussing it, you need not cite each and every single iteration. (This comes up a lot in citing particular dates or persons who’ve said “thing X” — you should only need to cite the original, the most thorough, or the most recent, when the most recent covers the intermediates towards the original.)
Citations only seem to be confusing when the reader is unfamiliar with the concept of what citations are for to begin with (and when one is so informed, one may be less confused), or when one is familiar but the reasoning for depicting such and such confusing: Why is person X citing that pseudo”scientist”?
Another point I’d like to bring up is about blog citations: If you are posting on your blog a photo of yours or art, and it is CC-BY, why should I cite the blog for the illustration, when I can just borrow the art, then attribute in the caption or acknowledgements that this comes courtesy Mike P. Taylor, PhD? It even takes less space. Blogs, on the other hand, are impermanent, vulnerable to hackers, and lack forms of “soft-copy” for the purpose of distribution. I’m beginning the process on my own blog to provide a pdf file for each and every single post (that’s 185 right now) but I lack the free space to reasonably sustain this to make “permanent” copies. What is the solution? Don’t cite but borrow and inform in the paper, or have someone else pay for secure storage.
August 8, 2012 at 1:01 am
If you are posting on your blog a photo of yours or art, and it is CC-BY, why should I cite the blog for the illustration, when I can just borrow the art, then attribute in the caption or acknowledgements that this comes courtesy Mike P. Taylor, PhD? It even takes less space.
I can think of at least two reasons: first, you might want to cite something besides an image, like an argument, in which case people will want a specific source, not just a mention of the author.
Second, so readers don’t have to go to two separate places (acknowledgments and references) to find all the references.
Blogs, on the other hand, are impermanent, vulnerable to hackers, and lack forms of “soft-copy” for the purpose of distribution.
Blogs are impermanent right now, but that is potentially fixable with a decent archiving system. That would solve the hacker problem, too, although to be realistic I have never heard of someone permanently losing any scientifically valuable blog post to hackers. The major blogging platforms are pretty robust and most have good Undo functions if something gets deleted. Lacking a soft copy for distribution only matters for the ICZN, and it’s shortly going to stop mattering even there.
In any case, citing blog posts is better than pers. obs., pers. comm., unpubl. data, and other link-to-nowhere citation modes that people already accept.
August 8, 2012 at 1:49 am
Even professional archiving systems are vulnerable to hackers.
August 8, 2012 at 12:25 pm
“what, when it comes right down to it, is peer-review for?”
It’s still for evaluating the “scientifically satisfactoriness” of work, no? It’s why we have confidence in journal papers and not oil-funded climate change denying white papers.
August 8, 2012 at 12:49 pm
Zed, I think that’s a pretty good answer.
But I suspect some difficulty arises because this evaluation is mixed in with the business of trying to help the author improve the paper.
August 8, 2012 at 2:54 pm
Matt, thanks for replying.
I can agree with you on the relative quality of citing blog posts, but should note that “pers. comm.” and “pers. obs.” arguments, or should be, removed. If anything, if someone feels they need to make such an argument, why can’t we just include the data in the first place, and remove this essentially un-verified claim from the literature before it becomes literature? Here, the “relative quality” I mention is that using blog posts for citing, and equating them with “pers. comm.” level statements is not useful to us in the least: We should be using primary data when at all possible. Perhaps part of the reasoning these statements were useful was because of limited paper space, but this is changing due to the development and increase in digital only publications or absence of page-costs to authors. Thus, the restriction on size or absence of quality on the argument is becoming a non-issue. Plopping in the data, when you are the originator, should be the first, not the second, option.
On the issue of the illustration though, my point there was only that you are already including the illustration in the paper — you do not need to cite a particular source to discuss it, when that material can be included or is identical to an argument you’ve already cited. This one follows the “citation padding” issue I mentioned.
Consider:
Should Mike Taylor, when redescribing Apatosaurus minimus, cite each and every photo that appears on this blog (or elsewhere when he’s posted it) prior to submission; or describe the material there in the first place? If he does both, which of these is redundant with the other? (That is, one of these can be excluded and provide the same information; which one do we toss to avoid cite-padding?) If one has something meaningful to say, why shouldn’t one include this description in the primary paper?
(I must apologize for the lack of peer-review oriented comments.)
August 8, 2012 at 2:55 pm
Jaime suggests:
One reason why you can’t always do this is if the person whose communication you’re quoting has a manuscript in prep or in review, and you don’t want to pre-empt it.
August 8, 2012 at 3:03 pm
But that means you are without data. Why are you referring to a topic you do not want to preempt? You’re already discussing it. Simply omit the reference, the citation (which isn’t one), and mention that “While research leads to this area, current research elsewhere by [Scientist X] is exploring an aspect of this work, and thus we do not explore this issue further here.” If the primary purpose of citation is to either reference data, or shorten explication, then a “pers. comm.” cite does neither. Moreover, it confuses the issue that the work may eventually change directions, and even end up concluding other than what the citing work implies it does (or lists a caveat the citer is either ignoring, or unaware of). The value seems less than supposition or hearsay, and for Science, that seems bad.
August 8, 2012 at 3:04 pm
No, of course not — not for either of the reasons that we cite. In describing the animal properly, I would not need to cite the less detailed blog posts as authorities; and I would not need to give credit to myself.
Ah, now that is another, and ultimately more important, question. If we were to publish the description here on SV-POW! rather than in a conventional journal, we really would be making an end-run around peer-review, and I don’t think that such a manoeuvre would be generally approved of. (For what it’s worth, it would also yield a new name that’s not valid according to the ICZN — not even under the proposed new electronic-publication rules, which require a journal with a DOI.)
But I don’t know whether, in ten years, that attitude is going to look amusingly antediluvian and hidebound. When you look at the purposes of publication, it seems to be that the gap between journals and blogs is still closing, and I can’t tell how close they’re going to get, or even if any gap will remain between them in the end. If this sounds ridiculous to you, remember how much suspicion electronic journals were view with only a decade ago — widely considered “not really real” compared with their paper-based brethren. Attitudes change. It’s hard to predict how.
August 8, 2012 at 3:13 pm
Maybe an example will help. In one of our manuscripts (the one that was just rejected, in fact), I and Matt wrote:
The Naish 2008 reference is to an SV-POW! article, Invading the postzyg, in which he illustrated the pneumatic site in question with a photo of “Angloposeidon”. The blog post discusses this in more detail than we needed for our argument. Citing the blog article both give credit for the observation where it belongs, and provided a link for people who are interested in the details to look them up. Seems to me like citing the blog post is the only reasonable thing to have done in this case.
August 8, 2012 at 3:36 pm
“Even professional archiving systems are vulnerable to hackers.”—I think Matt’s point is that if a blog were deleted by hackers it would likely be pretty easy to restore the articles.
“soft copy”—Y’all mean “hard copy”, right?
August 8, 2012 at 3:47 pm
Mike, I can agree that citing the blog post is a reasonable thing to do, in some cases. I would think perhaps that one should only do this for “introductory” or “discussion” remarks. Not when dealing with primary data. What potential solution would I offer you?
You are using a blog post to provide potential data that works to contradict an argument (ligaments in the intermetapophyseal trough), pointing to possible pneumatic structures in the trough. You cite personal work (photo cites), an external citation (Schwartz), and then a blog post. You are already working on MIWG 7306, so it is fair to suggest that you can include in your paper a photo of this bone, closeup of the region, and discussion. Moreover, you can include Darren in the paper, and simply attempt a broader discussion.
But that is a good example for what you are saying, in so far as you are trying to use the blog post as a form of explication. I might argue, instead, that blogs may not be the ideal place to include this information … but say rather that blogs are not ideal yet. Matt mentions that technology and archiving may eventually advance to the point that we may have permanent addresses for blog posts. We may advance to the point that we can submit blog posts to archiving sites where papers are housed. When this happens, but probably not before, I can accept your argument completely — that blog posts can and even should be cited.
August 8, 2012 at 3:52 pm
ech,
I mean “soft copy,” as in soft copy, with the likelihood for archiving. I agree with Mike and Matt on some other factors dealing with how publication is evolving, and how we should aim for inclusiveness of our digital era. We should note that some data may only be available in a digital format, and as such “soft copy” rather than “hard copy” (non-digital, print-based form) should be part of the equation.
August 8, 2012 at 3:57 pm
No. Matt and I are not working on MIWG 7306, and never have. Only Darren has, and he was not an author of this paper. Including the photo and adding him to the authorship would be completely disproportionate for such a relatively tiny contribution to a substantial paper. Citing prior work is definitely the thing to do; and in this case, the prior work happens to be in a blog post.
We can already do that, and in fact already have done so. I’ve archived SV-POW! posts both at WebCite and at Nature Precedings.
August 9, 2012 at 4:19 am
Mike, my apologies on confusing “the Archbishop” with your work.
I do not think “prior work” shoudl count blog posts without the recognition that establishes it. That said, who else is “archiving” their blogs? Who else is following the process on a public level? I wonder if, if it is only you and a few others, such effort is wasted in trying to validate “blogging” as “publication”? Right now, Twitter and Facebook have the same, if more more secure, archiving and “permanency” than blogs, but citing Twitter or Facebook as authoritarian?
My strongest criticism, I think, would be this: If blogging is legitimate enough to cite, for the purpose of having data and even ways to archive … then why are you trying to publish elsewhere? You could EASILY use this blog to “publish” any paper you want … so why aren’t you?
(Don’t get me wrong: I’m not trying to argue against your position, I am trying to discover a way to make the process of “blogging as publication” as acceptable as paper or digital journal publishing.)
August 10, 2012 at 5:43 am
Don’t get me wrong: I’m not trying to argue against your position, I am trying to discover a way to make the process of “blogging as publication” as acceptable as paper or digital journal publishing.
Interesting. I can’t speak for Mike, but I doubt if blogging will ever be as acceptable as journal publication. I just know that it’s demonstrably superior to some forms of citation that people take for granted now.
And I suppose I think that trying to make blogging as acceptable as journal publication is not only doomed but wrong-headed. There’s no need to enforce some kind of false equivalency. I think there’s room in the science publication ecosystem for little bolts of data or argument that are more substantial than pers. comms. or unpubl. data, but less substantial than journal articles. Blog posts don’t have to be as good as journal articles, they just have to be more than useless. And they are. So we should cite them, where appropriate.
August 10, 2012 at 9:14 am
Unsurprisingly, I agree entirely with Matt here.
Already, formal papers cite not only other formal papers but abstracts (which are much more insubstantial than the kind of blog posts we’re talking about here), manuscripts in press, in review and sometimes in prep, personal communications, and my absolute favourite “unpubl. data” (I’m talking to you, Stevens and Parrish 1999!)
The notion that all those are citeable and blog posts aren’t is unsustainable.
August 10, 2012 at 11:11 pm
The point I was trying to get at is what is the cut-off form of “publication” one SHOULD cite? It seems that the desire here is merely that the source be “informative” and in a form that can be verified without resort to secret information. What are the criteria by which a source may be cited? Please note that I am requesting an objective set of criteria, and that I am — and you should be — wary of conditions that permit certain strains of “publication” loose, but not others.
August 11, 2012 at 7:36 am
Well, this is pretty darned interesting. If I understand you right, you’re trying to define citeability inclusively, by finding the minimum criteria that something has to meet to be citeable. Whereas I’ve mostly been thinking in the other direction. In a world where pers. comm., pers. obs., and the dreaded unpubl. data are all acceptable modes of citation, the reverse question, “How low is the bar set for minimal citation acceptability?” can only be answered with “Pretty damned low.” Lower than blog posts, anyway.
I haven’t given much thought to finding or defining the lower bound, because I expect that the community will define it over time based on what people actually do. I have the strong feeling that this is the only rule that governs citation practice in the real world–what one thinks editors and reviewers will accept in terms of respectability. Over time, norms are established and one gets a feel for those norms by participating in the process. What’s interesting right now is that we’re in a state of flux, it’s probably going to get even weirder in the next few years, and it may be a while before new norms are established. Given that the entire ecosystem of scientific publication is experiencing accelerating change, it’s probably worthless to try to come up with precise definitions or boundaries. Even if one managed to capture what is acceptable right now, the answer will probably be different–maybe very different–in five years.
AND on top of that is the fact that different outlets have tended to have different rules about what counts and what doesn’t–some journals let you cite abstracts but not dissertations, some accept dissertations but not abstracts, some both, some neither, and so on. So there was no “one rule” even before the internet came along and made new modes of scientific communication possible.
So my answer to your question, “what is the cut-off form of “publication” one SHOULD cite?” is “No-one knows yet, except when defined by editorial fiat at individual outlets, but we’re all in the process of finding out.”
August 11, 2012 at 9:13 am
Also, I think that “what is the cut-off point for what’s acceptable to cite” is absolutely and utterly the wrong question. The fact that we’ve allowed ourselves to be trained by current practices to think is those terms is pretty stupid.
The real question is: what will make my current manuscript more useful? If citing a blog post, an in-prep manuscript or even a pers. comm. will make the manuscript more useful then I should cite it. If not, then I shouldn’t.
We are all too quick to lose sight of what all this is for. Our goal to to effectively communicate science in a form that is comprehensible, comprehensive, and capable of being reproduced. All practices that support that goal are good; all practices that undermine it are bad.
August 11, 2012 at 4:56 pm
I doubt we will find ourselves gradually refining standards on acceptability of citation. Two hundred years ago, it was acceptable for authors to not even cite at all: A “researcher” could use aphorisms and what was effectively hearsay and “pers. obs.” to tangentially refer to some topic in which that author did not want to go into detail.
Less than a hundred years later, some more unscrupulous paleontologists could refer to “specimens” that “confirmed” their arguments, and not one person would raise an eyebrow at these assertions. We depended on the argument from authority, not questioning the authenticity of the researcher because institutions such as the NHM (back when it was just the British Museum) or the MCZ or the Smithsonian might support that individual. It was de facto license for plagiarism, lying, and other forms of dishonesty.
Now? We get to merely explicitly state that a statement or object we refer to comes through in forms various, as “pers. comm.,” “pers. obs.”, etc. Despite the explicit labels, the practices are still there (certainly not as many of the faults so openly obvious).
I say we are NOT at the point where we can say we’ve not come a way long enough to think we are going to end up anywhere better. This issue is similar to the state of things when it comes to the ICZN: you can blindly follow the “tradition” as it has been practiced before, treating either the definitions of acceptable publication or (my own torch) taxonomy as methods which we must accept until “it changes” — whenever that is — or we can try to enact the changes ourselves. Be a force for compelling, rather than just trying to incite.
Set a standard: What criteria SHOULD we have to determine what is citable?
August 13, 2012 at 12:27 pm
Dear Mike,
I would like to depart from the discussion on your digression, and go back to peer-review.
First, I would guess the filter it provides is probably underestimated by anyone that does not see all the universally rejected papers (i.e. by everyone).
Second, it seems like in fact, it is not primarily the peer-review itself that bothers you, but the organization of publication by journals. They all receive more submission than they can publish, and most of them have to reject perfectly good papers. There are two easy (to state, not to enforce) answers to this. One could work with mega journals like PLoS ONE, without space limits, or one could organize a large number of journals together, so that a referee could say that the paper is not suitable for journal A, but would be good for journal B. One submission to many (or all?) journals at once (this is to happen in the forthcoming Forum of Mathematics). None of this solutions abandons peer-review (but they both twist it in different ways).
Third, on another subject. I cannot understand how you got yourself paleontologists in this weird situation where you format papers according to the journal it is submitted to *before acceptance*. In mathematics, papers are reviewed in the format of the author’s choice (the only mandatory thing is to provide a pdf file, as far as I know), and are formated according to the journal style only after acceptance. Moreover, this formating is supposed to be done by the journal itself (this is one of the reason we are supposed to be happy with publishers) except for the free or cheapest journals.
August 20, 2012 at 10:12 pm
I’m not so sure. We see all blog posts, after all, including those that would be rejected if there was a review system. And we just learn what’s worth reading and what isn’t. Better: my idea of what blog-posts are worth reading is probably different from yours, but since they’re all published, we can each choose the subset that we care about and that meets our quality standard. It’s not obvious to me why the same approach shouldn’t work for papers, too.
That is certainly a very big part of it. I’m not yet clear in my own mind exactly how all the issues fit together.
Ah, sanity! One more reason (arXiv is the main one) for biologists to envy mathematicians!
August 21, 2012 at 9:29 am
Concerning the filter issue, I think that the selection by journals make it possible for a scientist to get access to interesting papers from outside its own network. Without this filter, one would probably have to filter itself by authorship, or fame and rumors about what is to be read. This is already more or less the case, but journals browsing adds another entry for papers to reach readers.
The question is probably field dependent anyway. In mathematics, a paper contains the proofs, which are our equivalent for experiments. So understanding a paper completely provides certainty about the validity of its claims. But it is often a very difficult task, that can take weeks, or even years to achieve. Having a filter that lets through only papers which have been read and somewhat checked for seriousness gives us confidence in them (maybe it gives us too much confidence, actually). Even if real crackpots are easy to spot, I would be frightening to have somehow to review ourself thoroughly every paper we want to use in our research.
August 21, 2012 at 11:36 am
Frightening, yes; but it’s a paper ghost. No-one is suggesting such a future is desirable, and more importantly it’s impossible. Everyone knows that everyone else wants help with filtering. When everyone wants something, what you have is a market. And people, companies, non-profits and others will find ways to meet that need. There’s no peer-review filter for posting on blogs, yet you don’t have too much trouble finding the blogs you care about and that are worth your time, and ignoring the others. Better still, the filtering you do (maybe by what your friends tweet, or by what’s subscribed in your RSS reader) is customised to what you care about, not to what some reviewers and editors thought a “typical reader” would care about.
August 28, 2012 at 1:09 pm
Benoît’s mention of “One submission to many (or all?) journals at once” makes me wonder if this can (or does) work as a sort of auction, without detracting from the scientific value of the product or eating up researchers’ time excessively. Journals and authors both want to publish influential papers, or maximise the (short-, medium- or long-term) influence* of the papers they publish …I was going to list some other things that would imply different priorities, but then realised they were redundant or inessential: just details of market placement and historical burden.
So, is a single transparent market for scholarly publication just ahead? Is it inevitable, and is that actually a problem? I don’t pretend to have any idea, sorry.
[*’Influence’ may not be the perfect word here; I mean it to have a sign as well as magnitude, i.e. papers can have a bad influence, and that is bad. This may not be obvious to everyone – actually I know it’s not, so it needs to be said.]
But, how peer review would fit into any such system seems clearly to be quite arbitrary: pre-submission, post-acceptance, post-publication, or single or multiple rounds of review in between can probably all work fine, depending on the field, specific topic, or taste.
August 30, 2012 at 7:11 am
[…] (or rather, ineffectiveness) of peer review and the standard publishing system (here, here and here are the most recent entries), and this included a small discussion on the form and necessity of […]
September 5, 2012 at 2:21 pm
[…] has not seen any peer review, and as much as some of my colleagues hate peer review (here, here and here), it does sort out a lot of papers with major flaws. And it shows a lot of signs of being prepared […]
September 5, 2012 at 4:14 pm
Bingo.
Good to know!
September 22, 2012 at 7:01 pm
[…] mon post il y a quelques temps, et à une série de posts intéressants de Mike Taylor (posts 1, 2, 3), je me suis demandé à quel point les différences de point de vue sur l’expertise par les […]
September 25, 2012 at 9:55 am
[…] details of why my paper was rejected, and why I think they’re dumb reasons. In part three: what peer-review should actually be for, and what I plan to do with the paper […]
October 3, 2012 at 10:51 pm
[…] disillusionment with the traditional pre-publication peer-review process [post 1, post 2, post 3]. By coincidence, it was in between writing the second and third in that series of posts that I had […]
November 16, 2012 at 8:46 pm
[…] ONE is the single greatest thing to have happened to scholarly publication. Its approach to peer-review is precisely correct: if a submission is good science, it gets published, period. The journal makes no attempt to judge […]
November 26, 2012 at 12:31 pm
[…] part 1, part 2 and part 3 from a few months […]
January 21, 2014 at 8:55 am
[…] on peer-review (Where peer-review went wrong, Some more of peer-review’s greatest mistakes, What is this peer-review process anyway?, Well, that about wraps it up for peer-review), I likened peer-review to […]
March 25, 2014 at 9:48 pm
[…] (We discussed the distinction between those last two previously.) […]