April 10, 2013
Is there any justification for any of these practices other than tradition?
- Choosing titles that deliberately omit new taxon names.
- Slicing the manuscript to fit an arbitrary length limit.
- Squeezing the narrative into a fixed set of sections (Introduction, Methods, Results, Conclusion).
- Discarding or combining illustrations to avoid exceding an arbitrary count.
- Flattening illustrations to monochrome.
- Using passive instead of active voice (especially in singular: “we did this” may be acceptable but not “I did this” for some reason).
- Giving the taxonomic authority after first use of each formal name.
- Listing institutional abbreviations at end of the Introduction section, several pages into the paper.
- Using initials for names in the acknowledgements.
- Refusing to cite in-prep papers, dissertations and blogs (while accepting pers. comm.)
- Using numbered citations instead of Author+Date.
- Using journal abbreviations such as “J. Vertebr. Paleontol.” in the references.
- Formatting references
- Having references at all, rather than links.
- Putting figure captions and tables at end the end of the manuscript instead of where they occur.
- Arbitrarily relegating parts of the manuscript to Supplementary information.
- Submitting images in TIFF format (even for born-as-JPEG photos).
- Double-spacing manuscripts.
- Writing cover letters for submissions.
- Throwing away reviews once they’ve been handled.
- Allowing the final product to go behind a paywall.
Did I miss any?
April 3, 2013
Gah! No time, no time. I am overdue on some things, so this is a short pointer post, not the thorough breakdown this paper deserves. The short, short version: Schachner et al. (2013) is out in PeerJ, describing airflow in the lungs of Nile crocs, and showing how surprisingly birdlike croc lungs actually are. If you’re reading this, you’re probably aware of the papers by Colleen Farmer and Kent Sanders a couple of years ago describing unidirectional airflow in alligator lungs. Hang on to your hat, because this new work is even more surprising.
I care about this not only because dinosaurian respiration is near and dear to my heart but also because I was a reviewer on this paper, and I am extremely happy to say that Schachner et al. elected to publish the review history alongside the finished paper. I am also pleasantly surprised, because as you’ll see when you read the reviews and responses, the process was a little…tense. But it all worked out well in the end, with a beautiful, solid paper by Schachner et al., and a totally transparent review process available for the world to see. Kudos to Emma, John, and Colleen on a fantastic, important paper, and for opting for maximal transparency in publishing!
UPDATE the next morning: Today’s PeerJ Blog post is an interview with lead author Emma Schachner, where it emerges that open review was one of the major selling points of PeerJ for her:
Once I was made aware of the transparent peer review process, along with the fact that the journal is both open access and very inexpensive to publish in, I was completely sold. [...] The review process was fantastic. It was transparent and fast. The open review system allowed for direct communication between the authors and reviewers, generating a more refined final manuscript. I think that having open reviews is a great first step towards fixing the peer review system.
That post also links to this one, so now the link cycle is complete.
Schachner, E.R., Hutchinson, J.R., and Farmer, C.G. 2013. Pulmonary anatomy in the Nile crocodile and the evolution of unidirectional airflow in Archosauria. PeerJ 1:e60 http://dx.doi.org/10.7717/peerj.60
March 19, 2013
I find myself reading a lot recently about “portable peer-review” — posts like Take me as I am, and my paper as it is? by scicurious at Neurotic Physiology, which excellently diagnoses a terrible, wasteful problem in scientific publishing:
My papers don’t often get in with minor revisions. Often I’ve got a ridiculously puffed head about my own work (apparently), and send them to places which reject them out of hand, or suggest major revisions and piles of new experiments which we just cannot do for various reasons. Then the paper ends up shuttled around. Send it in, wait 3 months, get rejected. Reformat (+2 mo or even more depending on collaborators and how much other crap you’ve got on your plate at the time) and send it out again. Years go by. In the meantime, suggested reviewers begin to hate me and I run out of new ones (only so many people in the field!).
I really wish there was a way to get out of this. This sort of thing contributes to the long lag times and slowness of scientific advance.
What a waste! What a drag on the progress of science! What a ridiculous situation we’re got ourselves into, with our chasing-after-prestigious-journals games.
An inadequate solution
The solution proposed by scicurious is:
You submit a paper to a large umbrella of journals of several “tiers”. It goes out for review. The reviewers make their criticisms. Then they say “this paper is fine, but it’s not impactful enough for journal X unless major experiments A, B, and C are done. However, it could fit into journal Y with only experiment A, or into journal Z with only minor revisions”.
As an incremental improvement on the current system, this is good, if rather impractical to implement.
But it doesn’t go nearly far enough. It still wastes time by going to multiple journals, probably with different formatting requirements, requiring assessment (albeit more lightweight) by several editors. And it does all that in the name of getting a designer label onto the paper by placing it in a “good” journal.
What are we, fourteen?
High-school kids are dumb enough to judge other kids by how fashionable their clothes are, by the labels on them, by whether they’re the clothes other kids think are cool.
Have we really not got beyond that?
The ugly truth
Trying to get into “good” journals is an idiot game. (Notice I don’t say “an idiot’s game” — more on this distinction below.) Although the political and bean-counting value of getting into Nature is huge, the scientific value of getting into Nature is zero. A paper in Nature is literally no better at all than the same paper would be in PLOS ONE. (In fact, it’s probably less good, because it will be butchered to fit the draconian space requirements.) Spending time and effort in trying to get a given piece of research into Nature is just about the least useful thing that can be done for that research.
I think deep down everyone knows this. But of course scientists still waste innumerable hours formatting their work first for Nature, then for Science when it gets rejected, then for PNAS when it gets rejected again, and so on “down the ladder”. But that direction is only “down” by agreement. And the reason of course is because it’s widely (though not universally) believed that wearing these designer clothes is the way to get jobs and grants. That’s why people who are not idiots play this idiot game.
(Thanks heavens for funders and assessors who explicitly state that the journal a work is published in has no effect on how it’s evaluated. You can find such statements from The Wellcome Trust, and regarding the Research Excellence Framework (REF). I want to see more granting and evaluation bodies make similar statements, and I look forward to seeing a university hiring policy that says the same.)
A better way
Happily for me, I don’t need a job or a grant, so I have the luxury of standing on the sidelines, shaking my head sagely yet smugly at the ridiculous manouevres happening on the pitch.
I admit to my shame that I have played the getting-into-a-good-journal game in the past, just because I blindly copied what I saw my colleagues doing without really thinking about it. One result is that our neck-anatomy paper was needlessly held up for more than four years. No-one benefits from these delays. They are a completely avoidable net loss for science.
No more. I am done with having my work rejected for spurious (i.e. non-scientific reasons). I’m only planning on submitting to journals that don’t do that. I reject the idiot notion that the natural lifecycle of a piece of work involves multiple submissions-review-reject cycles. From now on, my cycle is: do some work, write it up, submit it, see it published, move on to the next thing.
And note that “move on to the next thing” is a crucial step here. What really burns me is not the four-year delays on the papers I mentioned above, but all the other work that I’ve not done because I’ve been buggering about, excuse my French, with the corpses of these long-dead projects instead of getting the next thing done. And if that’s true for me, I bet it’s true for you, as well. Yes, you, reading this!
As of now, except in exceptional circumstances, my plan is only to submit to venues where I know scientifically sound work will be accepted. That means “megajournals” like PLOS ONE, PeerJ and (I don’t know, I will look into it) maybe some or all BMC journals. It also means edited volumes that I’m invited to contribute to (though they have their own issues). It probably also means certain other journals, such as PalArch, though they don’t make it explicit (and it would be good if they did).
First clarification: to be clear, I am not arrogant enough to think this means I will never again have a paper rejected. No doubt there will be occasions where I’ve made significant scientific errors, and reviewers will have to point those out and recommend rejection. I don’t mind that: it’s peer-review actually doing its job, and I’d rather fix those mistakes before publication. What I’m done with is rejections on the basis of “not impacty enough for this journal”, or the often equally specious “not a good fit”.
Second clarification: I don’t absolutely rule out exceptions. There might be occasions where, say, an impact-selective journal announces plans to put out a special volume that I want to be part of. I might submit to that; then again, I might not. I’ll judge it as it comes. But the point is, any exceptions will be exceptions. When I start thinking “where shall I send this?”, my list won’t start with Palaeontology and JVP. I’m glad to have got those notches on my bedpost, but I don’t feel any great need to go back to them.
Third clarification: I do understand that others might not be in a position to make the same leap. I am 99.7% certain that Darren won’t, for example, as he is convinced of the absolute necessity of Science‘n’Nature papers to advance his career. Matt, on the other hand, can and I think will — he’s got a tenure-track job at a university that he likes, has no plans to move on, and doesn’t need “prestigious” papers for his tenure case, only good ones.
(It pained me to have to make that distinction. What a stupid world, where “prestigious papers” and “good papers” are not synonymous, and don’t even overlap that much.)
But for people who, like me, don’t need to have an eye on the possible job-power of “prestige”, it seems obviously better to do what advances science best and fastest. And what a tragedy that advancing science isn’t what gets jobs.
February 27, 2013
I was reading Stephen Curry’s excellent summary of Monday’s Royal Society’s conference on “Open access in the UK and what it means for scientific research”. One point that Stephen made is:
[David Willetts's] argument is that pursuance of green OA leads to an unstable situation in which the cancellation of subscriptions (because readers have free access) drains the system of the funds needed to manage peer review and other publishing costs.
As an analysis of the difficulties of Green OA, this is admirably precise. But my eye was caught by that phrase “funds needed to manage peer review and other publishing costs.”
I think we should make an effort to wean ourselves off the habit of talking about “managing peer review and other publishing costs”. We all recognise that publishers do not provide peer-review — we do. But it’s also true that publishers don’t manage peer-review, either. Once again, we do that, by acting as unpaid academic editors.
I know that this is not news. We all know this. But a habit of speech is affording publishers a degree of credit that their efforts don’t merit, and that clouds the debate. Let’s apportion credit where it belongs.
Of course there are still “other publishing costs”. These are real and not negligible (even though PeerJ’s financial model suggests they are much less than we have sometimes assumed). It’s right that we should acknowledge that there really are publishing costs; and that whatever financial model we end up will need to pay them somehow. But let’s make an effort to be more precise about what those publishing costs are. Managing peer-review is not one of them.
February 21, 2013
Matt and I were discussing “portable peer-review” services like Rubriq, and the conversation quickly wandered to the subject of PeerJ. Then I realised that that seems to be happening with all our conversations lately. Here’s a partial transcript.
Mike: I don’t see portable peer-review catching on. Who’s going to pay for it unless journals give an equal discount from APCs? And what journal is going to do that when they get the peer-review done for free anyway? If I was Elsevier, I wouldn’t say “OK, we’ll accept your external review and give you a $700 discount”, I’d charge the full $3000 and get two more free reviews done.
Plus, you know, I can get all the peer-review I want, free of charge, at PeerJ.
Matt: Yeah, that was pretty much my take. Even as I was sending that I thought about adding, “I wonder if this is one more thing that PeerJ will kill.” Only ‘abort’ is more the verb I want, in that I don’t see this ever getting off the ground anyway.
Mike: I think the world at large has yet to realise what a black hole PeerJ is, in the sense that it’s warping all the space near it. Pretty much every time I have any thought at all about scholarly publishing now, that thought it swiftly followed by “… or, wait, I should just use PeerJ for that.”
Matt: Exactly. It makes me think that we may be discovering the contours of that space-warping effect for some time, in that we’re used to one model, and that, among all the other things PeerJ does, it quacks something like that old model so we tend to think of it as a very cool duck, and not the freakin’ tyrannosaur that is going to eat scholarly publishing.
Also makes me think of that Paul Graham thing about noticing that the door is open, and there being a lag between the freedom to do something and the adoption of that newly facilitated action or behavior.
New thought: assuming PeerJ does not implode, will the established powers try to start PeerJ-alikes, and if so, what will they charge (amount), and what will they charge for (lifetime membership? decadal? annual? per 1000 pages published?).
Mike: Sweet metaphor. It’s true. It’s qualitatively different from other journals in two respects.
First, the APC is literally an order of magnitude less — and at that point, a quantitative difference becomes qualitative. Someone like [NAME REDACTED] would worry about paying $1350 to PLOS ONE, but didn’t even stop and think before saying, yeah, I’ll do that.
Second, the lifetime membership changes the game for all subsequent submissions. Now when you have a manuscript ready to go, your question isn’t going to be “where shall I send this?”, it’s going to be “is there are compelling reason not to send this to PeerJ?”
Legacy publishers won’t start PeerJ-alikes because they can’t. As noted in many SV-POW! posts, Elsevier takes about $5000 for each article they put behind a paywall. Slice away the 40% profit and you get $3000 which not coincidentally is what they charge as an APC. They have old, slow, encumbered systems and processes and top-heavy organisation. At $3000 they are only breaking even. They can’t compete at a PLOS-like $1350 level and they can’t even think about competing at PeerJ levels. If they offered a lifetime membership they’d have to ask $10k or something stupid.
I don’t think it’s that they don’t want to change. They can’t. They’ve ossified into 1990s companies running on 1990s software. It’s hard to steer a ship with a $2bn turnover, and impossible to replace the engines while still cruising.
Matt: I think it is probably a mistake to think that PeerJ will only encroach “upward”, onto the territory of more traditional journals (which is “all of them”). We’ve already talked about it taking business from arXiv (at least ours, although there is the large non-overlap in their respective subject domains–for now, anyway).
But my point is, the question, “Why wouldn’t I send this to PeerJ?” may not only kick in for papers that you might conceivably send elsewhere, but also for manuscripts that you might not conceivably send anywhere.
Matt: Right. And if one is on the fence, shove it on the PeerJ preprint server and see what people have to say.
Mike: I think it’s the first megajournal to have an associated preprint server, and that may yet prove the most important of all its innovations.
Matt: It feels almost … struggling to find the right word, in part because it’s late and I need to go sleep. “Seditious” is not quite it, and neither is “seductive”.
At that point we started talking about something else, so I never did find out what word Matt was groping for. But what’s only gradually become clear to us is how much PeerJ is changing how we think about the academic publishing process. It’s shaking us out of mental ruts that we didn’t even know we were in. Exciting.
Big news yesterday. Identical bills were introduced into the US House of Representatives and Senate that, if passed, will make federally-funded research freely available within six months of publication. Here’s the exact wording, from the press release on Mike Doyle’s (D-PA) website:
The Fair Access to Science and Technology Research Act (FASTR) would require federal agencies with annual extramural research budgets of $100 million or more to provide the public with online access to research manuscripts stemming from funded research no later than six months after publication in a peer-reviewed journal.
As Peter Suber explains here and here, FASTR is a stronger version of FRPAA, the Federal Research Public Access Act, which has been introduced in Congress three times before (2006, 2009, and 2012) but never come up for a vote. However, momentum for open access is gathering, both on the supply side with progressive new outlets like eLife and PeerJ, and on the demand side of, well, citizens demanding access to the research they’ve already paid for, and legislators increasingly agreeing with them. So FASTR has a real shot at getting to a vote, and if voted on, could well pass. Which would be awesome, because we all need access.
I am especially happy that FASTR has bipartisan sponsorship in both houses of Congress. The sponsoring representatives in the House are Mike Doyle (D-PA), Kevin Yoder (R-KS), and Zoe Lofgren (D-CA). The identical Senate bill was introduced by John Cornyn (R-TX) and Ron Wyden (D-OR). So we’ve got Democrats from deeply blue states and Republicans from deeply red states, which is awesome and totally appropriate, because this issue really does cut across party lines. And, hell, last year Elsevier managed to hire bipartisan sponsorship for their toxic–in more ways than one–and rapidly-killed Research Works Act, so it’s nicely symmetrical that politicians from both sides of the aisle have come together to sponsor that bill’s near-opposite.
What can you do? If you live in the US, contact your legislators and tell them to support FASTR! It takes almost no time at all and it makes a big difference. This afternoon I called all five of the sponsoring legislators to thank them, and I called my representative and both California senators to encourage them to support the bill, and all told it took just a little over half an hour. If you skipped the thank yous and just got in touch with the legislators who represent you, it could be done in 15 minutes, and you’ve probably wasted more time than that today daydreaming about dinosaurs. Here’s what you’ll need.
Encourage your legislators:
Thank the bills’ sponsors:
- Senator John Cornyn (R-TX): (202) 224-2934
- Senator Ron Wyden (D-OR): (202) 224-5244
- Representative Mike Doyle (D-PA): (202) 225-2135
- Representative Zoe Lofgren (D-CA): (202) 225-3072
- Representative Kevin Yoder (R-KS): (202) 225-2865
This is big. This matters. Send an email, pick up the phone, make a difference.
I didn’t have any really motivational “contact your legislators!” artwork so the photos in this post are of papier mache dinosaurs–all stinkin’ theropods, I’m afraid–that I’m building with my son. More to come on that soon, but in the meantime, check this out and give it a whirl–after you contact your legislators!
February 14, 2013
There are a lot of things to love about PeerJ, which of course is why we sent our neck-anatomy paper there. I’ll discuss another time how its pricing scheme changes everything for Gold OA in the sciences, and maybe another time write about how well its papers display on mobile devices, or about the quick turnaround or 21st-century graphical design of the PDFs.
But among the most interesting things about PeerJ is its use of open peer review: reviewers are encouraged (though not required) to disclose their identity, and authors are encouraged (but also not required) to make the review history publicly available along with the final papers.
Uptake of open peer-review
Uptake of this option on the initial batch of 30 papers has been OK: 12 papers (40%) have had reviews posted:
(Articles 4, 18, 20, 23, 24, 32 and 35 do not exist — presumably they didn’t make it through review, typesetting and proofing in time for the launch. Or maybe they were rejected after having been assigned numbers.)
It’s interesting to see that most of the earliest papers did elect to publish reviews, but few of the later ones. This may reflect that the “early adopters” — the people who were quickest to get their submissions in after PeerJ opened its doors — also tend to be the more open-oriented people in other respects. It would be great if the authors of some of those other 18 papers were to make their reviews open, too: I’m sure it’s not too late.
What’s the value of open peer-review?
First, it improves transparency. In standard peer-review, three people (and editor and two reviewers) make a decision on behalf of the entire community, and no-one else can see what was done or why. In our case, John Hutchinson was our handling editor. We’ve often said on this blog how much we like and respect him, and it would be easy for someone on the outside to suspect that he’d been tempted to give us an easy ride. Anyone who reads the review history can see for themselves that he didn’t.
Second, it gives credit where it’s due. Reviewers who do a good job often plough in many hours of time that they could be spending on their own work, and it’s right that they should be recognised. In this case, Heinrich Mallison did a careful line-by-line critique of the whole 50-page manuscript and sent up a marked-up copy which was invaluable in making revisions. That sort of work should be acknowledged. [At the moment, that marked-up manuscript is not on the PeerJ review-history page. I've been told they're going to fix that.]
Third, it gives blame where it’s due. Some reviewers who are excessively critical, or criticise in a non-constructive way that can’t be addressed in a revision; others are positive about the manuscript but make no real contribution to improve it. It’s right that reviewers who don’t do their job properly should be called out on that. (Of course anonymity can go some way towards shielding bad reviewers, but even then it’s often quite obvious who’s responsible for a given review.)
Fourth, it encourages good behaviour from reviewers. When they know their good work will receive credit and their bad work will reflect on them, they will have more incentive to do their best. Too often, reviews are seen as a tax on researchers’ time. Making them visible helps to bring them into the mainstream.
Fifth, it avoids wasted effort. Sometimes a review is a serious piece of work in its own right — Matt tells me that for one manuscript we was refereeing, he wrote a detailed critical review that was longer than the manuscript itself. Of course, no-one ever saw that work but the original author and his handling editor, which is a terrible waste. Publishing reviews fixes that.
Sixth, and this is crucial, open peer-review is a fantastic teaching tool. Matt has already explained how showing his Western students real reviews in a real process is going to help them much more than made up ones.
What are the drawbacks of open peer-review?
Search me. I sure as heck can’t think of any.
Changing peer-review culture
PeerJ didn’t invent open peer-review — far from it. It’s been around for a while, practiced by some BMC journals and also adopted more recently by eLIFE — another of the new breed of born-digital open-access journals. Another new publishing initiative, F1000 Research, is built entirely on the concept of open review.
The importance of PeerJ doing the same is that it helps to bring open peer-review into the mainstream. PeerJ’s going to be a big journal — its explicit goal is to be a PLOS ONE-scale megajournal. One of the many things it can achieve is to help shift the default reviewing culture to open.
January 29, 2013
A separate concern is whether the OA business model is sustainable in the long term of decades or even centuries. By contract, OA content has almost no commercial value, unless it is re-published in a for-profit volume. How confident can we be that the content of an OA journal that goes bankrupt will be preserved in an openly accessible way?
Don’t worry — you can be very confident. Reputable open-access journals arrange for their content to be archived in well-trusted third-party archives such as PubMed Central and CLOCKSS. See for example PeerJ’s blog about the arrangements they’re making or this statement from PLOS ONE.
A much more serious problem is this: what happens to the content of a non-OA journal when it goes bankrupt? In general, copyright for the content of such journals is owned by the publisher. This not only means that informal archive arrangements such as BioTorrents and The Disks Of Millions can’t be used — worse, it means that content archived in PubMed Central or CLOCKSS may never become available. If a failing publisher sells its assets, that will include the copyrights — and since literally any unethical corporation might sniff an asset-stripping opportunity, that could be disastrous.
In short, you can be much more confident that PLOS’s content will still be around in 10, 20 and 100 years than you can that Elsevier’s will.
January 17, 2013
My new article is up at the Guardian. This time, I have taken off the Conciliatory Hat, and I’m saying it how I honestly believe it is: publishing your science behind a paywall is immoral. And the reasons we use to persuade ourselves it’s acceptable really don’t hold up.
Because for all that we rightly talk about the financial efficiencies of open access, when it comes right down to it OA is primarily a moral, or if you prefer idealogical, issue. It’s not really about saving money, though that’s a welcome side-effect. It’s about doing what’s right.
I’m expecting some kick-back on this one. Fire away; I’ll enjoy the discussion.
December 23, 2012
After the authors’ own work, the biggest contribution to a published paper is the reviews provided, gratis, by peers. When peer-review works as it’s supposed to, they add significant value to the final paper. But the actual reviews are never seen by anyone except the authors and the handling editor.
This is bad for several reasons.
First, good reviewers don’t get the credit they deserve. That’s unfair on those who do a good job — who generously invest a lot of time and effort in others’ work.
Second, bad reviewers don’t get the blame they deserve. That leaves them free to act in bad faith: blocking papers by people they don’t like, or whose work is critical of their own; or just doing a completely inadequate job. Because there are no negative consequences for doing a bad job, just people have no external incentive to straighten up and fly right.
Third, the effort that goes into reviewing is largely wasted. Often the reviews themselves are significant pieces of work (that’s certainly true when I’m the one giving the review) and the wider community could benefit from seeing them. Frequently reviews contain extended discussion, not only of the paper’s subject matter but of scientific philosophy such as approaches to taxonomy or narrative structure.
Fourth, editors’ decisions remain unexplained. Most editors handle manucripts efficiently and fairly, but there are cases when this isn’t the case — as for example when I was one of three reviewers who wholeheartedly recommended acceptance but the editor rejected the paper. Even discussing that situation was difficult, because the reviews in question were not available for the world to read.
Fifth, and more general than any of the above, the reviewing process is opaque to the world. In times past, logistical reasons such as lack of space in printed journals meant that the sausage-machine approach to the review process was the only feasible one: no-one wants to see what goes into the machine or what goes on inside, we only want the final product. But we like in an increasingly open world, and consensus is that pretty much all processes benefit from openness.
There are various initiatives under way to change the legacy system of reviewing, including F1000 Research and the eLife decision-letter system. But at the moment only a small minority of papers are submitted to such venues.
What to do about the others?
And so I found myself wondering … what would happen if I just unilaterally posted the reviews I receive? I already make pages on this site for each of my published papers (example): it would be easy to extend those pages by also adding:
- The submitted version of the manuscript
- All the reviews I received
- The editor’s decision letter
- My response letter to the editor
- The final published paper.
I know this is “not done”. My question is: why not? Is there an actual reason, other than inertia? Wouldn’t we all be better off if this was standard operating procedure?
[Note that this is orthogonal to reviewer anonymity. As it happens, I think that is also a bad thing, but it's independent of what I'm proposing here. I could post an unsigned review as-is, without revealing who wrote it even if I knew.]