Counting beans
October 10, 2012
The reason most of my work is in the form of journal articles is that I didn’t know there were other ways to communicate. Now that I know that there are other and in some ways demonstrably better ways (arXiv, etc.), my enthusiasm for sending stuff to journals is flagging. Whereas before I was happy to do it and the tenure beans were a happy side-effect, now I can see that the tenure beans are in fact shackles preventing me from taking a better path.
Sauropod Vertebra Picture of the Week 
October 10, 2012 at 6:29 am
Are you suggesting leaving the journal article system altogether? Physicists have not surpassed us by ditching journals – they’ve simply surpassed us by being better at posting pre-prints prior to formal publication. We can all agree that much of the problem in paleontology is with time to broad availability. Initial release of preprints, followed by a formal, peer-reviewed version of record, gets the best of both worlds. I posit that rather than a wholesale escape to *publishing* on arXiv and blogs, a broader use of pre-prints is needed in paleontology (for which the previously mentioned venues can suffice). Folks can start to use the work, and any needed corrections can be made in the meantime. Given the choice between a non-reviewed preprint and a reviewed version of record, I’ll go with the latter every single time.
October 10, 2012 at 6:35 am
I still think peer review is more positive than negative, and you don’t get that in the same way with arXiv or blogposts.
At this moment in time, I think you run the risk of your work being marginalized and not read if you largely eschew peer-reviewed journals. There are plenty of workers who I think would be reluctant to cite a blog post or arXiv pre-print, prefering to stick to venues that are considered acceptable by the field at large (i.e., a journal). I’m not necessarily advocating that position, just saying a significant number of folks still think that way.
The peer-review and editorial processes are far from perfect, but I think we’d have to wade through a lot more sh*tty science if they didn’t exist.
October 10, 2012 at 6:35 am
Frightening, isn’t it? Matt and I discuss these kinds of issues a lot offline — what gets posted on SV-POW! is only a summary of the most coherent 10% — and we are both very much still figuring out where we stand. I’ll say as well that both of us have shifted our positions hugely in the last 12-24 months; neither of us is really clear even now on what we think; and in so far as we do have clear positions we don’t fully agree.
But I do share with Matt the sense that the current journal system is incredibly wasteful: it absorbs far too much time that should be spent doing actual science, and it retards publication by a ridiculous amount. Exactly how those issues would be best fixed is hard to think through. But we’re trying.
October 10, 2012 at 6:40 am
That is certainly what I would have said a couple of months ago. I’m strongly of the opinion that posting our neck-anatomy paper on arXiv was the best thing to do, and I would unhesitatingly do it again. And we did it with every expectation of following up with a peer-reviewed publication in a more conventional venue.
It’s just that now when I contemplate that process, I genuinely question whether my time best benefits science if I invest it in yet another huge editorial pass on work that is essentially complete (and which, remember, has already benefitted from the feedback of SIX reviewers in total); or whether it’s better than I invest that time in new work instead.
And “question” really is the word: my mind is very far from made up. The thing to bear in mind here is that the choice is not between (A) bothering to go through the peer-review-and-publication process, and (B) not bothering — which is how we usually portray it. The choice is between (A) bothering to go through the peer-review-and-publication process, and (B) doing something else instead.
I am really keen to hear a lot of opinions on this.
October 10, 2012 at 7:14 am
Mike,
I think the sauropod necks paper is a bit of a red herring, because in that case, its a choice between (A) bothering to go through the peer-review process for a third time, and (B) doing something else instead. But going to purely preprints & blogposts with future manuscripts is a conscious decision to avoid the formal peer-review process for even the first time.
October 10, 2012 at 7:22 am
A system that could fit you would be to devise mega-overlay journal, to be understood as follows.
An overlay journal is a journal that does not archive nor disseminate article: they are posted in preprint version on an open archive (e.g. arXiv), then submitted via their ID number, and if accepted the possibly revised definite paper is posted on the archive only as a new version, and linked to by the journal website (note that one can either drop the typesetting and copy editing, or perform it and ask the author to post the corrected version, both are possible). So, the journal becomes simply an entity that organizes peer-review and lists the accepted papers on a web site.
A mega-(electronic)-journal can have a different view than usual journals on selectivity: without backlog nor size issues, it can publish all publishable papers (in a non-pejorative sense: all papers that are correct and advance our understanding of the field).
A mega-journal (in the above sense) can also have highlighted articles selected from its published article by suggestion of the editorial board or referees. This can achieve two goals: readers can skim all papers in their field and all highlighted papers around their field to get a sense of what is going on; and people have more incentive to submit important papers than when they feel they will be buried in “lesser woks”.
If a field has two or three journals that are both overlays and mega, then most good papers can be submitted only once and published. It is better to have several choices, first because monopoly gives bad incentives to the editorial board, second because one can always be unfairly rejected by referees and should have another submitting option.
In this system, we keep the good points of both the preprints and peer-review. Note also that if a journal drops copy editing and relies on author-prepared manuscripts, then it can run with much fewer costs, and therefore at much lesser rates than even PLoS ONE. If an institution or set of institutions are willing to build one, they mainly need a few assistants and a good software (Open Journal System and EasyChair are good option that alreqdy exist) and can tun in Diamond OA (=neither the author nor the reader pay anything).
October 10, 2012 at 7:24 am
I think I still agree with that. Or at least, I think I do where PLOS ONE-like reviewing is concerned, filtering for whether something is good science rather than for sexiness, fashionability or alignment with a particular community’s interests. The latter aspect is responsible for maybe half of the pain of peer-review, and for perhaps 2% of its value. At the moment my best judgement is that the pain of the former is probably worth more than the benefit, the pain of the latter absolutely is not, and the pain of the whole package … I don’t know. Obviously it varies.
I am surprised you would equate arXiv with blog-posts. Although SV-POW! posts have been cited in more formal papers, it’s not something I necessarily expect to see a lot of (and most instances of that so far have been in papers by Matt or me). But something in arXiv is very different: it’s written as a scientific manuscript rather than a popular article, and it’s archived securely and persistently in an immutable form. Personally I wouldn’t hesitate for a moment before citing something from arXiv, whereas I would at least stop and think before citing an article from a blog that I personally don’t maintain. And for what it’s worth, there is simply no issue at all in the maths and physics communities about citing arXiv papers.
I do agree that this is a danger. We are out on the edge of a moving platform right now, and we can’t really tell what direction it’s going to take or how fast it’s going to move. Even if I was fully convinced in my own mind that (say) publishing only on arXiv was just as good for the progress of science as publishing in journals, then I would probably keep publishing in journals anyway, just so that I’m not seen as some nutter on the fringes. (And we all know there are plenty of those hovering around on the outskirts of palaeo.)
So that is one to continue publishing in journals. Another, as Matt notes, is so that tenure committees can unthinkingly count the publications without needing to agonise over What Counts.
To me, that makes no sense from anyone who’s prepared to cite a pers. comm. — the citation that is guaranteed not to point to anything. Why would we ever accept those and not blog citations? For myself, my only reservation about citing a blog post would be fear that it would go away. Which is why I happily cite SV-POW! (which I know will not) and of course arXiv.
Sure.
Well, we would certainly need to ignore a lot more sh*tty science.
But I find myself wondering whether that really matters. Already I ignore a million papers a year. Does it really make a difference if I have to ignore two million instead?
October 10, 2012 at 7:26 am
Randy’s last message does make a good point. A distinction that hadn’t occurred to me.
October 10, 2012 at 7:31 am
I do like the idea of overlay journals. They seem like an excellent solution to several problems. What I’d like to know is why, so far, there don’t seem to be any. Is it a problem of bootstrapping enough prestige to make it worth people’s while to work on them? Maybe it needs a couple of really senior people to kick one off.
Or do they already exist in maths and physics?
October 10, 2012 at 7:56 am
Teach the controversy! Certainly science journalists, lawmakers etc. will have no problem figuring out which papers are good and which to ignore when you, Dave Peters, and Duane Gish are publishing side-by-side without the shackles of review oversight.
October 10, 2012 at 8:29 am
You make a good point, CY. A few weeks back, in one of my more cynical moods, I suggested that the principle value of peer-review is that it provides an opportunity for authors to demonstrate that they are prepared to undergo peer-review. Maybe that really is a benefit.
That said, don’t forget that Dave Peters has had his work published in the Journal of Vertebrate Paleontology, which is about as mainstream as our field gets.
October 10, 2012 at 1:34 pm
I’m an astronomer, so I’ve been using arXiv for over a decade. It does a lot of things really well. I submitted my most recent paper to the arXiv at the same time that I submitted it to a journal for peer review, because in this particular case, we were reporting on an object which would disappear in a few months. In situations where time is important, arXiv beats journals with a big stick.
I have reservations about arXiv from time to time, because the lack of peer review means that cranks can submit papers with really crazy ideas (and there are a LOT of people with crazy ideas about cosmology, let me tell you ….). But I’ve been surprised at the relatively small number of such papers which have appeared in arXiv over the years. I’m not sure exactly why, but it’s an encouraging sign.
Why aren’t there any overlay journals? Maybe the idea is just too new, as technology has moved faster than editorial boards. But there may be a financial reason: what’s the profit model in creating such a journal? If scientists adopt the arXiv model in a big way, to the point that tenure committees count the arXiv papers as well as other papers, then who will pay for the overlay journal to read old news? If scientists don’t adopt the arXiv model in a big way, then will there be enough “hot” papers to support the price of an overlay journal?
I do like one aspect of current dead-tree journals: there’s a very good chance that a person will be able to access the information in them 100 years from now. I’m much less confident that a person will be able to access the information in arXiv 100 years from now. And I have very little confidence that a person will be able to access the excellent SV-POW archives 100 years from now. Mike wrote:
I certainly hope that SV-POW will not go away, but I wonder: who runs the servers which host it? Who pays for the servers? I know that some science blogs are hosted by corporations, for example. Can someone explain the situation for SV-POW? Is there a backup plan, for example?
October 10, 2012 at 2:44 pm
Re: Mike’s comment – don’t forget that some workers I won’t mention have also had their work _rejected_ from many journals too (for very good reason)! Peer review worked.
October 10, 2012 at 3:00 pm
220Mya says: “I still think peer review is more positive than negative, and you don’t get that in the same way with arXiv or blogposts.”
You do get that with blog posts, though — comments! In fact, comments are better in a way, because anyone can make them at any time, and anyone can see them.
October 10, 2012 at 3:09 pm
Perhaps the value of the journal system then is that it *forces* peer review? Selected reviewers – and not just friends or trolls of the blog owners – are required to provide feedback. I don’t think anyone would claim that blog comments are unbiased in a way that peer review is not.
October 10, 2012 at 3:20 pm
Michael Richmond asks:
The server is run by wordpress.com at no cost to us. Against the possibility that this will someday change, I periodically download an XML dump of the entire site (a facility that WordPress offers). This archive is stored on both of my own computers, plus a server in Denmark, plus a mirror in the USA, plus offsite backups of the server.
In short, if WordPress.com went away, there would for sure be an interruption in the service; but we’d certainly be able to restore the site from backups. (A proper service such as arXiv is obviously better set up for hot failover.)
In the mean time, since the whole site is CC-BY, we encourage anyone with concerns to download their own copies and mirror it at the locations of their choice.
October 10, 2012 at 4:16 pm
Thanks, everyone, for the comments.
First, for those of you who are worried that I’m going to put on a tinfoil hat and start looking for ancient astronauts, fear not. I didn’t say that I was going to abandon the journal system. The reality is more depressing: I can’t afford to abandon the current journal system. I still need those tenure beans.
Second, I’m not alone in thinking that there is better way to do this. Michael Eisen:
Go read the whole post, it’s short and awesome. What will publish-then-filter communication look like for the sciences? In practice, probably not too different from blogging.
CY suggests that without the imprimatur of peer review, people won’t be able to tell my work apart from that of David Peters and Duane Gish. This is actually a really important question. Is my work well-regarded because:
A. it’s good work, with accurate observations and reasonable inferences;
B. I wrote it;
C. it’s subject to peer review
D. some other factor I haven’t thought of yet.
It would be nice to think that it’s A. alone, but it ain’t. For one thing, no one is perfect, everyone lets a stinker through now and then, and we can’t read everything that comes out and evaluate it on its merits. We need some signals that guide us to good work and steer us away from BS without having to read and evaluate every single paper first.
It can’t be C. alone. David Peters’ pterosaurs-are-prolacertiforms paper appeared in a peer-reviewed journal. So has almost all of the work of the BANDits.
It shouldn’t be B. alone, otherwise we either have a cult of personality or the argument from authority. That said, reputations are important, and one reason they’re important is that they become a shorthand for quality of work (A). CY’s selection of bogeymen is telling. David Peters and Duane Gish have both earned reputations for pushing bunk. Where they pushed the bunk is much less important than the fact that it’s bunk.
I realize these factors are not independent. If I have a decent reputation (B; at least to some!) it’s because I’ve published quality work (A; he said modestly) that I subjected to peer review (C). So as Mike said, one of the major benefits of peer review is that it is a signal of the author’s seriousness and willingness to be corrected.
But wait. Everything I put up here on SV-POW! is subject to correction, and by a vastly larger pool of reviewers than I’ve ever had for a published paper. And, as Mike Keesey pointed out, all of these post-publication reviews (i.e., blog comments) are publicly visible and subject to correction themselves, unlike typical peer reviews which never see the light of day. Isn’t that better?
Andy: “I don’t think anyone would claim that blog comments are unbiased in a way that peer review is not.” I’ll see your unbiased and raise you publicly visible and correctable. Are unbiased comments that are sometimes arbitrary, sometimes wrong, rarely subject to correction, and always secret better than biased comments that are out in the open where they can be seen and evaluated by all? I’m not asking that rhetorically, I genuinely want to know what people think. I wonder how much of the fickle nonsense of the current peer review system would go away if the reviews were signed and published along with the papers–which will undoubtedly be the model in any publish-then-filter system like that advocated by Michael Eisen.
Fire back, this is one of the most productive and probably most important discussions we’ve had on this blog.
October 10, 2012 at 5:00 pm
Comments on a few statements:
I don’t equate them, but I did include them in the same sentence, because you and Matt have both mentioned these as a form of publication that can be cited. They are different in many ways, but they are both pre-peer review publication avenues you have utilized. And in fact, Matt included in this very post a link to a scientific paper that was published on a blog. So Matt associated the two, not me.
No one said traditional peer review was perfect. All journals publish some crap, but peer review can provide a basic filter (provided its PLoS One style – focusing on the science, not ‘impact’). And even a broken clock is right twice a day – meaning that just because one worker publishes alot of crap doesn’t mean some of their work isn’t useful. But hopefully peer review helps increase the usefulness vs. crap ratio.
But its not the same, because there’s no guarantee you’ll get comments on a blog post, or that those comments will in any way equate to true peer review. For example, though I know many gainfully employed vertebrate paleontologists (i.e., tenure-track faculty and curators) read SV-POW!, only a small minority actually ever leave comments. Similarly, I applaud PLoS One for trying to make it easier for community peer-review to occur by making comments on a published paper; however, as many folks have pointed out, the vast majority of their articles have zero or one comment. That’s not effective peer review.
This is an extension of Mike Keesey’s point. But I maintain that quality is more important than quantity. At risk of insulting your readers, getting lots of comments on a post is not necessarily enough – what is the quality of the comments, and what is their source? Twenty comments from interested avocationalists are not necessarily the same as a review from a peer who specializes in the same field as you do. I know that sounds elitist; I’m not saying that an avocationalist can’t give a good and helpful review, just that there’s a background of knowledge about the subject that makes specialist reviews so useful. If you publish on a blog, there’s no guarantee you’ll get review or commentary from a researcher who specializes in sauropods, neck anatomy, etc., for example.
October 10, 2012 at 5:11 pm
Re: Matt’s comment: I completely agree that publicly visible and correctable reviews are *generally* more desirable – but this was not my main point. As Randy re-stated, where the peer review process usually works better than blogs is that it *forces* “experts” to review the paper. No such guarantee for blog comments. This does not mean that blog comments are bad, just that they have a potential shortcoming in this way versus peer review. Ideally, I think we want a system where reviews are public, correctable, and actively solicited from appropriate experts (with potential for those not solicited to have their say too). Neither blogs nor most journals have achieved this yet.
October 10, 2012 at 6:25 pm
Ideally, I think we want a system where reviews are public, correctable, and actively solicited from appropriate experts (with potential for those not solicited to have their say too). Neither blogs nor most journals have achieved this yet.
I agree, on both points: that would be ideal, and we’re not there yet. Do any journals publish reviews alongside papers?
October 10, 2012 at 6:36 pm
I seem to remember that some PLOS journals may have done this way back in the day for some articles, but a quick search couldn’t find any specific examples.
October 10, 2012 at 6:43 pm
Yes, my recollection was correct, according to this on the PLOS ONE site.
October 10, 2012 at 6:51 pm
Thanks to all who have commented on what seems to be the emerging key point here: in what ways and to what extend do blog comments resemble, differ from, fall short of or surpass traditional pre-publication peer-review.
[Side-note: I didn’t even realise this was one of the key issues until this discussion got started. One of the most exhilarating things about running this blog is that much of the time, I genuinely don’t know what I think until I find out when I start to write about it in response to other people’s comments.]
I think the question has been elucidated quite well as follows: blog comments win by being quick, openly visible, non-anonymous, numerous, subject to correction; but traditional reviews win by being thorough and detailed, provided by experts, and obligatory: they are solicited by an editor and the reviewers feel an obligation to come through.
Rather than ask which form of review is superior, I think we can ask a better question: is there a way to combine to good qualities of both?
And I think the answer is yes. The approach would be to begin by posting the submitted manuscript immediately, before review. At that point, an editor would solicit peers to provide detailed reviews of the posted manuscript. The reviews themselves would then be posted alongside the initial manuscript, attributed to their authors. The manuscript and the reviews would be subject to further, unsolicited, comments — like blog comments — which would also be posted publicly with attribution. Finally, the authors would revise the manuscript in accordance with the solicited reviews, also taking into account any unsolicited comments they consider useful, until the editor is satisfied. At that point, the final revised manuscript would also be posted, and stamped as “peer-reviewed”.
So the final posted object would consist of initial manuscript, two or more solicited reviews by peers, zero or more (maybe many) unsolicited comments from peers and others, and a final accepted manuscript. The work will have been made available as quickly as possible, will have benefitted from review by peers and comments from others, and the reviews will themselves be subject to correction from other parties.
As Andy points out, “neither blogs nor most journals have achieved this yet”. I think that perhaps PeerJ, when it launches, may be the closest thing we have. Are there others?
October 10, 2012 at 10:27 pm
For examples of journals that have both post-publication and public review (still with option to be anonymous though) take a look at the journals of the EGU like Biogeosciences and Ocean Science.
October 11, 2012 at 9:15 am
Thanks for the pointer, Elisabeth. EGU Biogeosciences is at http://www.biogeosciences.net/9/issue1.html
It looks like a good journal — truly open access (CC BY 3.0), and as Elisabeth said publishing the reviews and the authors’ responses to them. Why haven’t I heard of it before?
The only downside is the service charges, which are paid on submission and definitely on the high side for long papers — it would have cost us €3527.16 =~ £2830 or $4535 to submit our recent neck-anatomy manuscript (76 manuscript pages × €39 per page for MS-Word format not using their template, plus 19% VAT). I see no mention of fee waivers, so unfortunately this journal would be a non-starter for someone like myself without institutional funding.
October 11, 2012 at 9:30 am
Liz Wager made a good point, back in 2009, on a BMJ blog entitled If comment is cheap why is peer review so expensive?:
It’s true that the batch-processing approach taken by traditional peer-review makes it feel antagonistic and confrontational. I can imagine back-and-forth working much better.
Come to think of that, I don’t even have to imagine it — I’ve experienced it. One of my papers went through several rounds of (we thought at the time) hostile review, to the point where I and my co-authors seriously considered pulling it from the journal and taking it elsewhere. Somewhat in desperation, I set up a Skype-video chat with the handling editor, who was also the most difficult reviewer. Almost immediately, most of the difficulty went away. Given a face to talk with, it was easier to understand what were the key points of his best criticisms, and also easier to explain the ways in which we felt some of his other criticisms were in error. The paper was published, shortly thereafter, in a form that both he and we were happy with.
October 11, 2012 at 9:35 am
I guess those journals are in different field and we all get a bit field blinded with time.
They do have a financial support program (http://www.biogeosciences.net/general_information/financial_support.html), but it’s a bit of a pain to find it though as the link is light grey (says Financial support). And it doesn’t take you to the page it promises anymore for some reason… Here’s an excerpt of it though:
“EGU Service Charges Waivers
The EGU waives the service charges for publications in their journals with an amount of at least 10% of the previous year’s page budget to be assigned to authors lacking support of funds. After a manuscript has been accepted for publication, the authors may contact the Copernicus Publications Editorial Support (editorial@copernicus.org) to apply for partial or full APC waivers, which will be granted subject to support by the handling editor and approval by the respective chief editor. In special cases, such as long review manuscripts, the authors may contact the chief editor prior to submission.”
October 11, 2012 at 10:12 am
You could easily save €900 just by using their template. It’s not exactly very restrictive. Looking at your ~100000 character manuscript on Arxiv, I don’t see why you couldn’t. LaTeX it and you’d save another €450! Still a lot of money if one for some reason cannot get the fee waiver, but just to show that there are ways of cutting the costs from the highest category.
October 11, 2012 at 12:15 pm
Another issue to consider for the “blog-like” model of scientific publication is the nature of the comments which may be made. I enjoy reading a number of blogs which present material in a serious manner — for example NPR’s Planet Money. There are usually many comments after each article, but many of them — sometimes more than half — are politically-motivated rants which have little to do with the topic at hand. It’s certainly discouraging to someone who is interested in asking questions and receiving answers about the material in the post.
Now, those off-topic comments are probably more common in blogs which discuss political issues (as the economic-minded Planet Money sometimes does), but they may very well appear in some scientific circles as well. Traditional journals tend to publish very few letters from readers, if any, and of course the editors can simply ignore anything which is irrelevant. But how can we prevent off-topic trolls, or even well-meaning and on-topic trolls, from dominating the comments in an open forum?
October 11, 2012 at 12:40 pm
Elisabeth writes:
No; I could save €900, but it wouldn’t be easy. Re-templating a manuscript of that size is not a trivial undertaking. I probably would do it if I were going to subbmit to Biogeosciences, because €900 is a lot of money. But I wouldn’t be happy about it.
And worse:
That’s only a saving if my time is free, which is very much isn’t. Transforming that manuscript into LaTeX would take long enough that I’d effectively be working for less than minimum wage. Not a good use of resources. Worse, once the manuscript was in LaTeX format, it would be effectively useless for pretty much any other palaeo journal, should I not get past review at Biogeosciences.
Note that this isn’t a criticism of the journal: I’m not saying that their prices are unfair or exploitative, of that the different price scales for different formats are unreasonable. I’m just saying that for me, with my manuscript, they add up to either a lot of money or a lot of work. And when I invest a lot of work, I want to making new science, not doing drone-level clerical tasks.
October 11, 2012 at 12:54 pm
Michael Richmond makes an important point:
True. You have only to look at the Youtube video of your choice, or any palaeo-related story in a national newspaper, to see how dishearteningly stupid comment threads can be.
In this respect, it’s possible that I’ve been misled by my experience here on SV-POW! and over on my other blog, The Reinvigorated Programmer. Both blogs have well-established (though always changing) communities of commenters, who pretty much without exception contribute real insight through discussions in the comments — as this very post demonstrates nicely! In five years, we’ve only once had to shut down a comment thread, we’ve never banned anyone, and I can’t offhand think of a single occasion where we’ve had to delete a comment. That is despite not moderating — except for the first comment by each new commenter, and comments containing many links.
I suppose the question then becomes this: if we do transition to a world where blog-like comments on published articles become common, would we expect them to be more like the comments at SV-POW! or those at the Daily Mail?
October 11, 2012 at 8:37 pm
I for one think Mike’s combined pre- and post-review format for publication would be ideal.
I do somewhat off-topically want to object to Matt’s statement “It can’t be C. alone. David Peters’ pterosaurs-are-prolacertiforms paper appeared in a peer-reviewed journal. So has almost all of the work of the BANDits.” though. While Peters’ more recent ideas are often problematic, his 2000 paper was at least as good as those by respected professionals on the same topic, like Hone and Benton (2008). Comparing it to BANDits’ work is simply unfair.
October 12, 2012 at 8:08 am
Don’t forget that the Dave Peters JVP paper that I linked to earlier in the thread was his 2009 one. Now I know nothing about pteroid articulation so I have no idea whether it’s a good paper or not. We can certainly hope that is is good, and that it made it through peer-review on the basis of its content rather then being rejected on the basis of its authorship.
October 12, 2012 at 2:59 pm
I do somewhat off-topically want to object to Matt’s statement “It can’t be C. alone. David Peters’ pterosaurs-are-prolacertiforms paper appeared in a peer-reviewed journal. So has almost all of the work of the BANDits.” though. While Peters’ more recent ideas are often problematic, his 2000 paper was at least as good as those by respected professionals on the same topic, like Hone and Benton (2008). Comparing it to BANDits’ work is simply unfair.
I think you need to go back and look at Peters (2000) again, particularly the points where he has photographs and interpretive drawings adjacent to each other. The figures purporting to show complex antorbital fenestrae in prolacertiforms are about as unconvincing as anything I have seen in the published literature. And there are features linking pterosaurs to dinosaurs and other archosaurs that are downplayed or just flat ignored. So we’ve got overenthusiastic image interpretation passing for key pieces of data, a counting-the-hits-and-ignoring-the-misses style of analysis, and lots of special pleading. Sounds like a BANDit paper to me.
October 14, 2012 at 6:48 pm
[…] merits of formally published papers and more informal publications such as blog-posts a couple of times, but perhaps never really dug into what the differences are between […]
October 16, 2012 at 9:03 pm
Perhaps this just displays my naïveté, but I’m shocked to find (a) that your papers aren’t composed in LaTeX to begin with, and (b) that paleo journals don’t take and use LaTeX submissions. Isn’t “paleo” supposed to be the subject, not the method?
October 16, 2012 at 9:17 pm
Har! For myself, I’d be perfectly happy to use LaTeX: I spent a decade writing troff, I’ve written a shedload of raw HTML and far too much DocBook-schema XML, and I’ve written the odd computer-science paper in LaTeX along the way, so text formats with inline processing instructions don’t scare me. But no-one uses anything like this in palaeo. I don’t think I’ve ever seen, or been involved with, a submission in any format other than Microsoft Word. I’m not saying that’s a good thing — very far from it! — only that it’s true.
October 16, 2012 at 10:11 pm
My spouse (who has a Ph.D. in physics) is similarly puzzled by the lack of LaTeX in paleontology (although I’m puzzled by the fact that a statistics class isn’t required on their end, so I guess we’re even). Were I to speculate, I would say it’s because there is not as much of a “programming culture” in paleontology versus other fields like physics or engineering. Things like mark-up look scary, I guess! I predict that may change with more and more paleontologists becoming comfortable with R and other fancy tools.
October 17, 2012 at 12:25 am
Elizabeth wrote:
“Some journals let you see the other reviewers’ comments, but usually several weeks or months later. Reviewers sometimes see the authors’ response but this usually feels as if it were snarled through gritted teeth and any expressions of thanks to the reviewer usually seem formulaic and flattering. Wouldn’t it be nice if we could have a real dialogue?”
To which Mike responded:
“It’s true that the batch-processing approach taken by traditional peer-review makes it feel antagonistic and confrontational. I can imagine back-and-forth working much better.”
Back and forth communication is, I think, the best part of the work, in total. When you are in school, you are encouraged to interact with your advisor, and in many cases the project for your Master’s and Doctorate are done in conjunction with teachers and fellow students; no one does a project wholly alone. At the same time, when a manuscript is prepared, you have it draft reviewed by friends and colleagues, followed by submission and the “savage” effects of peer review.
These are two different levels of review, and indeed they can be combined as we’ve had discussions here. The first is from people who are familiar with your work, and may be involved in your personal life in some way; the second are people who are themselves potential competitors, who are put into a deliberative adversarial function: find what’s wrong, make them fix it, or suggest it not work no matter what.
It is my intention to have my drafts reviewed deliberately by people who will disagree with me, but my purpose is not antagonism but discussion: I not only want to know what is wrong, I want to know why, and how. Perhaps I will learn something new in the process. It is one reason I leave many of my discussions short, because I want others to bring discussion to the floor — but it doesn’t always work. My favorite posts on my blog are my jaw-orientation ones (because they are deeply technical and yet very personal), while my most remarked upon ones are about soft-tissue reconstruction and Spinosaurus spines, and the discussions on these are very, very different, or nonexistent.
I like the adversarial system, however, because we cannot always rely on our friends to be utterly honest with us. If we will have our feelings hurt after a few rounds of review, what happens when a sharp-tongued comment from one of our friends? Will they feel safe enough to make that comment? It is important that we not try to inoculate ourselves from what we feel we don’t need to hear, and instead agree only on what we like, the pleasant review versus the harsh criticism.
Problematically, blogs endorse the level of power one can have over filtering review, so that post-publish review doesn’t seem very … effective. I can ignore what you say, just as you ignore what I say. If instead, we pretend that a paper is a paper, and the product is the point, then the time it takes to finish, the rewriting needed to fix it, and the balls to admit you erred and not take it personally, then things will be better, regardless of the medium.
October 17, 2012 at 6:58 am
You make a good point, Jaime, that the adversarial position adopted by some peer-reviewers can be helpful. It’s for this reason that the original “devil’s advocates” were invented: lawyers appointed by the Catholic Church to argue against the canonisation of a new saint. The idea is that an argument for something is only seen to be strong if it’s come through the best possible argument against it.
So that suggests that the problem with peer-review as it currently stands may not so much be that an adversarial position is adopted, but that it’s done at a distance so that none of that helpful back-and-forth is possible. (Also, the distance insulates Jerk Reviewers from seeing the consequences of their Jerk Actions.) Something like a dissertation defence might work better as a means of improving the paper.
October 17, 2012 at 2:57 pm
Jaime, you raise some good points, but you have a very idealized vision of how peer review works. For example:
It is my intention to have my drafts reviewed deliberately by people who will disagree with me, but my purpose is not antagonism but discussion: I not only want to know what is wrong, I want to know why, and how. Perhaps I will learn something new in the process.
Yeah, that would be fantastic. If every review worked like that, Mike and I would still be focusing on whether review should take place after initial publication (so it doesn’t slow down the dissemination of the data) or before, but we wouldn’t be whingeing about what a crapshoot it is.
But it is a crapshoot. Just because something goes out for review does not mean that it is going to get a tough but fair review that both identifies weaknesses with the goal of making the paper as good as it can be (not overly positive), and does so without ripping the paper, or even the author, to shreds (not overly negative).
Now, I am not going to argue that my papers have not been improved by peer review. Almost all of them have been, to some degree. But in some cases I’ve gotten one tough-but-fair review and a second review that was basically a blank pass, or a broad-spectrum slagging without any specific recommendations for improvement. The former have helped some papers sail through review without as much critical attention as they deserved, and the latter have often prevented papers from being accepted at all, sometimes on completely spurious grounds.
Every time I send a paper in, I recommend Jerry Harris as a reviewer. Not because he’s a friend, but because I know that he will go over the paper with a fine-toothed comb and catch almost every factual error and weak argument, but in a constructive fashion that aims to make the paper better. If I could get Jerry Harris–or Leon Claessens, or Paul Barrett*–as a reviewer every time, or, more realistically, if I could rely on getting reviews from others of the same quality, I’d have fewer problems with the system. Not no problems, but fewer problems.
* Before anyone gets their feelings hurt, I’m not saying that these three are the only ones I’ve gotten tough but fair reviews from. But when I think about the reviews I’ve gotten that most improved the work, these three names come immediately to mind.
October 17, 2012 at 3:01 pm
Sadly, it really is. There are some reviewers who I can’t imagine ever recommending acceptance for anything I submit, and others who I can’t imagine recommending rejection. So by far the biggest determining factor of whether my submissions make it through peer-review is who the editor happens to select.
That’s crappy.
March 15, 2013 at 7:36 pm
[…] blog, we went to the trouble of publishing it in a journal. Especially in light of sentiments like this. For my part, it’s down to two things. First, to paraphrase C.S. Lewis, what I wrote in that […]
November 6, 2014 at 8:06 am
[…] When it comes to evaluation by a job-search panel, or a promotion committee, or a tenure board, what counts? And that is a very hard question to answer, as it depends largely on the institution in question, […]