Last Friday I got an email from Dr Stuart Taylor, Commercial Director of the Royal Society, wanting to set up a phone-call to talk about the issue I raised about the editorial procedure on Biology Letters. I got back to him with my Skype handle, but without fixing a date or time.

Then on Monday this week I was approached by Lucas Brouwers, a journalist for the Dutch daily newspaper NRC Handelsblad. He was writing a story about Biology Letters‘s misrepresentation of reviewer judgements and submission-to-acceptance times, and wanted me to answer some questions.

I thought it would be best to talk to Stuart Taylor before sending my answers to the questions, so my responses could take into account whatever he had to say, so I got back in touch with him and brought our phone-call forward. As it happened, he called me just as I was about to hit Send on my half-composed reply to Brouwers, so that was good timing.

I asked him if it would be OK to recount the main points of our conversations; he assented, so here there are:

  • First, he was very polite and friendly — graciously so, really, in light of the things I’ve said about the Royal Society, and the boycott that Matt and I have initiated.
  • Second, he was insistent that the use of the revised manuscript’s submission date in the the published PDFs is not intended to game statistics. I accept that most individuals at the Royal Society and its journals don’t have that motivation in mind.
  • Third, I pointed out, and he conceded, that it nevertheless does have the effect of apparently reducing the submission-to-acceptance time and apparently increasing the rejection rate. I still find it hard to imagine how this convention got started if not from someone‘s desire to game the statistics.
  • Fourth, Stuart Taylor made the point that many other “high-prestige” journals do what Biology Letters is doing. I have no patience with this defence, and told him so: “all the other kids are doing it” cut no ice with my mother when I was growing up, and nor should it with us. Even if it’s true, I return to the point that we’d hope and expect the Royal Society to set a higher standard of integrity, not to sink to the less honest level of other publishers.
  • Fifth, there is a meeting of the editorial board at the end of the month, and Stuart Taylor intends to raise this issue at that meeting.
  • Sixth, I made the point that it won’t take much to fix this: instead of just stating Received and Accepted dates on the first page of published papers, the journal could give three dates: Received, Revised and Accepted. I got the impression, though I don’t remember any clear statement, that Stuart Taylor thought that was a reasonable thing to do.

Overall, I am pretty happy that the society seems to be taking this fairly seriously. It is serious, but it’s also very easy to fix. I hope they’ll just do it, and we can move on.

Meanwhile, the Dutch newspaper article was published, with quotes from both me and Stuart Taylor. It’s in Dutch, so I am not sure what it says exactly. Here is the text of the article. If anyone wants to provide a translation I’d be grateful. (I mean a proper translation, not just shoving it into Google Translate.) [Update: Lucas Brouwers has himself provided a translation now.]

Blad ‘sjoemelt’ met datering

Door onze redacteur
LUCAS BROUWERS

ROTTERDAM. Volgens twee onderzoekers sjoemelt het veel gelezen wetenschappelijke tijdschrift Biology Letters met de datering van de artikelen die het publiceert.

Vaak wordt een ingediend artikel in eerste instantie afgewezen. Dan herschrijven de auteurs het, zij dienen het opnieuw in, waarna het artikel alsnog wordt geaccepteerd. In dat geval vermeldt Biology Letters alleen de datum van de tweede indiening. Daardoor lijkt de omlooptijd van artikelen kleiner dan hij daadwerkelijk is. Het tijdschrift wordt zo aantrekkelijker voor wetenschappers die snel willen publiceren.

Aanleiding voor de aantijging vormt de publicatie van een artikel over langnek-dinosauriërs, vorige week woensdag. Paleontoloog Mike Taylor van de Universiteit van Bristol ontving het manuscript op 17 juli ter beoordeling als peer reviewer. Taylor was positief, maar Biology Letters wees het artikel toch af en vroeg de auteurs het te herschrijven en opnieuw in te sturen. Dat gebeurde op 21 augustus. Deze datum werd vervolgens op het gepubliceerde artikel als datum van indiening vermeld.

Taylor en een collega kondigden daarom afgelopen vrijdag een boycot af. De twee zullen geen artikelen meer beoordelen voor de Britse Royal Society, waar Biology Letters deel van uitmaakt, totdat het genootschap zijn beleid verandert. Volgens Taylor is de oplossing simpel. “Vermeld de ware indieningsdatum, en het probleem is verholpen.”

Stuart Taylor (geen familie), commercieel directeur van de Royal Society, is het niet met de lezing van de twee onderzoekers eens. “Mike Taylor impliceert dat wij dit doen om onze omloopcijfers beter te doen lijken”, zegt hij aan de telefoon. “Dat bestrijd ik, alhoewel ons beleid inderdaad dat gevolg heeft.”

Op een redactievergadering zal Stuart Taylor de kwestie bespreken. Hij acht het niet ondenkbaar dat de tijdschriften van de Royal Society in de toekomst inderdaad de oorspronkelijke indieningsdatum zullen vermelden. “Wij hebben niets te verbergen.” Volgens Stuart Taylor is het afwijzen van artikelen, om vervolgens een aangepaste versie te accepteren gemeengoed onder wetenschappelijke tijdschriften.

Auteur Nicole Klein (universiteit van Bonn) betreurt het dat haar artikel onderwerp van discussie is geworden. “In onze ervaring verloopt het publicatieproces bij Biology Letters snel en professioneel.”

Finally, anyone who’s been following this issue should read this lengthy comment from Paul Barrett, who among his other roles is on the Editorial Board of Biology Letters.

Further bulletins as events warrant.

Advertisements

Counting beans

October 10, 2012

The reason most of my work is in the form of journal articles is that I didn’t know there were other ways to communicate. Now that I know that there are other and in some ways demonstrably better ways (arXiv, etc.), my enthusiasm for sending stuff to journals is flagging. Whereas before I was happy to do it and the tenure beans were a happy side-effect, now I can see that the tenure beans are in fact shackles preventing me from taking a better path.

Folks,

In response to our recent post about reject-when-you-mean-revise and submission-date massaging at Royal Society journals, Susie Maidment tweeted:

Since then I have heard from several other sources — including Stuart Taylor, Head of Publishing and Commercial Director of the Royal Society — that these practices are widespread.

Can anyone confirm this from their own experience?

It needs to be stamped out wherever it happens.

Just a quick one for Matt Butler, who in a comment on the orignal postwrote:

I just looked on the BL website, and the only infromation I could find was “Articles submitted to Biology Letters benefit from its broad scope and readership, dedicated media promotion and we aim for a turnaround time of within 4 weeks to first decision.” This sounds like a much more reasonable claim to make and one that would be harder to fudge.

I just looked as well, and here’s what I saw:

Front page of Biology Letters web-site, http://rsbl.royalsocietypublishing.org/, as of 7:12am on Saturday 6 October 2012

So there it is, prominently displayed right on the front page:

Average receipt to acceptance time: 28 days

So it’s not just that false submission-to-acceptance dates are given on individual papers; but the false average is used as a promotional tool.

If you haven’t already read the last post, please go do so before reading this one. Please also see this response from the editorial office of Biology Letters.

I’ve discussed all of what follows with Mike and in this post I speak for both of us.

To briefly recap: like many (but not all) journals, Biology Letters and at least some other Royal Society journals publish dates of manuscript submission and acceptance with each published paper (see an example at the bottom of the first page of this recent paper). Submission-to-publication dates are ostensibly useful for establishing priority in the case of research groups knowingly or unknowingly competing, and for potential authors and others who want to compare the turnaround times at various journals, especially journals like Biology Letters that pride themselves on rapid turnaround. In both cases, the date of interest is the date of initial submission of a manuscript. And since the submission dates quoted by Biology Letters are identified only as “Received [date]”, one would normally assume that that represents the date of initial submission.

But it does not.

At a typical journal a paper deemed worthy of publication by the reviewers would be sent back to the authors for revision, and if the revised manuscript is approved by the editor it may be accepted for publication in the journal. The sequence of events is the same at Biology Letters, but the labels are changed. In the case currently under discussion, a paper that three reviewers recommended for publication (“as it is”, “in essentially its current form”, and with a recommended change to a single paragraph) was rejected, and the rejection letter to the authors made the blatantly false claim that the rejection “has been taken on the advice of referees, who have recommended that substantial revisions are necessary.” That’s a lie, plain and simple.

The authors did “resubmit”–meaning that they performed the minimal revisions and sent the manuscript back, which at any typical journal would not be considered a new submission, and the paper was published by the journal, but with the “Received” date being that of the return of the revised manuscript, not the initial submission. This fudging of the numbers massively shortens the perceived turnaround time of the journal–in the current case, by 60%. As Mike succinctly put it in the last post, “We have the journal lying to authors about the status of their manuscripts so that it can then lie to the readers about its turnaround times. That’s deeply screwed up.”

The editorial office of Biology Letters has now responded to explain their policy. Here’s the relevant section in full:

We receive a huge number of submissions to Biology Letters and competition is intense, with the vast majority of papers being rejected. Therefore, we provide the option of ‘reject and invite resubmission’ to provide editors with the facility to invite resubmission of papers which show great promise but are not publishable in their present form. We believe that authors value this option – it is better than outright rejection and having to start the process again with another journal. The paper is technically a new submission so the time starts again.

So, two problems here. First, they’re taking something that’s clearly negative–lying to authors about their manuscript status so they can fudge their turnaround times–and presenting it as if it is a positive thing. As if the only alternative is to reject any paper that isn’t completely perfect the first time with no possibility of resubmission. As if almost all journals don’t allow authors to revise their work and send it back in without the revised manuscript being considered a new submission. As if they’re not doing the exact same thing that all other journals do in practice, but changing the meaning of the words to make themselves look better–and more to the point, faster. So in the case of the Klein et al. paper, “reject” actually means “accept essentially as-is”, “on the advice of referees” means “in direct contradiction of the advice of the referees”, “received” means “whenever we got the revised version back that we intended to publish”, and “a new submission” means “a resubmission of revised manuscript that changed very little from the initial submission”.

And this from the Royal Society, part of the same nation that produced Orwell.

Second problem is the last line of the section quoted above: “The paper is technically a new submission so the time starts again.” Well, let’s unpack that. In this case “technically” means using the new, counter-intuitive and indeed counter-factual definitions of the terms given above–or what in common parlance we would again call lying. And “so the time starts again” is incomplete; what they obviously mean but do not go on to say is “so the time starts again so that our perceived turnaround time can be as short as 40% of the actual turnaround time at a journal that didn’t fudge its numbers.”

Or, if we accept the obvious fiction that the revised manuscript is “a new submission”, then it would seem that this newly submitted manuscript gets accepted without peer review. As Mickey Mortimer explained in a comment on the previous post:

If the resubmitted paper “is technically a new submission” so that it deserves a new ‘submitted on’ date, shouldn’t it also be peer reviewed again by experts? If the peer reviewers’ original comments are really all that is needed, then surely the new paper can’t be that different from the old one, and should thus have the old submission date attached. If the new paper was actually substantially different, I’d say it hasn’t been properly peer reviewed.

So, either Biology Letters is publishing newly submitted manuscripts without having put them through peer review (what they rather bizarrely claim to have done), or they’re just lying by referring to revised manuscripts as “new submissions” (what they’ve actually done). They can’t have it both ways.

All in all, this response from the Biology Letters editorial office is about the most empty piece of writing I have seen outside of politics or postmodernism. If you dig through the banal reassuring verbiage, you find that they just restate their crimes in terms that sound positive, like when the US Department of Defense used to describe radiation exposure in “sunshine units“. I am disappointed–by the specific treatment of Klein et al. in the case of the current paper, in the discovery that this is not an isolated incident but is in fact journal policy, and in the pathetically transparent non-response of the editorial office.

What to do, what to do.

In looking back over my notes, I see that 12 of the 64 peer reviews I have performed to date in my career have been for Royal Society journals, either Proceedings B or Biology Letters. In fact, in 2009 I was named one of their “Top Reviewers” because of the number of reviews I performed that year. They sent me a certificate and everything. Of Mike’s 16 peer reviews, 4 have been for Royal Society journals. So we are stakeholders in this process. We invest considerable time and mental energy in reading manuscripts and providing the most thoughtful and constructive reviews that we can. The Royal Society benefits from this unpaid effort, in that editors get our expert opinions on which manuscript are acceptable for publication, and authors–the actual lifeblood of any journal–get feedback that hopefully makes their work better.

We’re not going to continue to invest that effort in an organization that has a deliberate policy of lying to authors about reviews and lying to everyone about the speed of manuscript turnaround. Until the Royal Society adopts more honest editorial policies, we will no longer perform peer reviews for papers submitted to Royal Society journals. We encourage anyone who currently reviews for Royal Society journals to join us. If you’re an author considering sending your work to a Royal Society journal, ask yourself whether you want to have your paper spuriously and dishonestly rejected just so the journal can fudge its numbers. If you’re a reviewer, ask yourself whether you want your laboriously constructed report to be ignored by the editor, even when it agrees with those of the other reviewers.

Members of the Biology Letters editorial office, and members of the Royal Society at large, we welcome your comments.

I’ve recently written about my increasing disillusionment with the traditional pre-publication peer-review process [post 1, post 2, post 3]. By coincidence, it was in between writing the second and third in that series of posts that I had another negative peer-review experience — this time from the other side of the fence — which has left me even more ambivalent about the way we do things.

On 17 July I was asked to review a paper for Biology Letters. Having established that it was to be published as open access, I agreed, was sent the manuscript, and two days later sent a response that recommended acceptance after only minor revision. Eleven days later, I was sent a copy of the editor’s decision — a message that included all three reviewers’ comments. I can summarise those reviewers’ comments by directly quoting as follows:

Revewer 1: “It is good to have this data published with good histological images. I have only minor comments – I think the ms should generally be accepted as it is.”

Reviewer 2 (that’s me): “This is a strong paper that brings an important new insight into a long-running palaeobiological issue […] and should be published in essentially its current form.”

Reviewer 3: “This manuscript reports exciting results regarding sauropod biomechanics […] The only significant addition I feel necessary is to the concluding paragraph.”

So imagine my surprise when the decision letter said:

I am writing to inform you that your manuscript […] has been rejected for publication in Biology Letters.

This action has been taken on the advice of referees, who have recommended that substantial revisions are necessary. With this in mind we would like to invite a resubmission, provided the comments of the referees are taken into account. This is not a provisional acceptance.

The resubmission will be treated as a new manuscript.

I can’t begin to imagine how they turned three “accept with very minor revisions” reviews into  “your manuscript has been rejected … on the advice of referees, who have recommended that substantial revisions are necessary”.

In fact, let’s dump the “I can’t imagine how” euphemism and say it how it is: “reviewers recommended substantial revisions” is an outright lie. The reviewers recommended no such thing. The rejection can only be because it’s what the editor wanted to do in spite of the reviewers’ comments not because of them. It left me wondering why I bothered to waste my time offering them an opinion that they were only ever going to ignore.

Then six days ago I heard from the lead author, who had just had a revised version of the same manuscript accepted. (It had not come back to me for review, as the editor had said would happen with any resubmission).

The author wrote to me:

The paper will be published (open access) at the 3rd of Octobre. When I had submitted the corrected version of the ms acceptance was only a formality. So [name] was right, they just want to keep time between submission and publishing date short.

Well. We have a word for this. We call it “lying”. When the editor wrote “your manuscript […] has been rejected for publication in Biology Letters … With this in mind we would like to invite a resubmission … This is not a provisional acceptance. The resubmission will be treated as a new manuscript”, what she really meant was “your manuscript […] has been provisionally accepted, please sent a revision. The resubmission will not be treated as a new manuscript”.

I find this lack of honesty disturbing.

Because we’re not talking here about some shady, obscure little third-world publisher that no-one’s ever heard of with fictional people on the editorial board. We’re talking about the Royal Freaking Society of London. We’re talking about a journal (Biology Letters) that was calved off a journal (Proceedings B) that emerged from the oldest continuously published academic journal in the world (Philosophical Transactions). We’re talking about nearly three and a half centuries of academic heritage.

And they’re lying to us about their publication process.

When did they get the idea that this was acceptable?

And what else are they lying to us about? Can we trust (for example) that when editors or members submit papers, they are subjected to the same degree of rigorous filtering as every other submission? I would have assumed that, yes, of course they do. But I just don’t know any more.

Sampled specimens, sampling locations and cross sections of sauropod cervical ribs. (a) Anterior neck of Brachiosaurus brancai (Museum für Naturkunde, Berlin) with hyperelongated and overlapping cervical ribs. (b) Three cross sections were taken along the proximal part of the posterior process of a left mid-neck cervical rib of Mamenchisaurus sp. (SIPB 597) in ventral view. Note the medially pointed ventral part of the cervical rib. (c) Seven cross sections were taken along the left ninth cervical rib of B. brancai (MB.R.2181.90), which is figured in lateral view. (d) Neck of Diplodocus carnegi (cast in the Museum für Naturkunde, Berlin) with short cervical ribs. (e) Six cross sections were taken along the right mid-neck cervical rib of cf. Diplodocus sp. (Sauriermuseum Aathal, Aathal HQ2), which is figured in ventral view. Note the morphological differences of this cervical rib when compared with the hyperelongated cervical rib of B. brancai. (Klein et al. 2012:figure 1)

The paper in question is Klein et al.’s (2012) histological study confirming that the bony cervical ribs of sauropods are, as we suspected, ossified tendons — as we assumed in our recently arXiv’d sauropod-neck paper. I am delighted to be able to say that it is freely available. At the bottom of the first page, it says “Received 21 August 2012; Accepted 13 September 2012”, for a submission-to-acceptance time of 23 days. But I know that the initial submission — and remember, the final published version is essentially identical to that initial submission — was made before 17 July, because that’s when I was asked to provide a peer-review. Honest reporting would give a submission-to-acceptance time of 58 days, which is two and a half times as long as the claimed figure.

Now the only reason for a journal to report dates of submission and acceptance at all is to convey the speed of turnaround, and lying about that turnaround time completely removes any utility those numbers might have. It would be better to not report them at all than to fudge the data.

This is another way that the high-impact fast-turnaround publishing system is so ridiculously gamed that it actually hurts science. We have the journal lying to authors about the status of their manuscripts so that it can then lie to the readers about its turnaround times. That’s deeply screwed up. And it’s hard for authors to blow the whistle — they don’t want to alienate the journals and the editors who have some veto power over their tenure beans, and reviewers don’t usually have all the information. The obvious solution is to make the peer-review process more open, and to make editorial decisions more transparent.

That, really, is only what we’d respect from the Royal Society. Isn’t it?

Note. Nicole Klein did not know I was going to post about this. I want to make that clear so that no-one at the Royal Society thinks that she or any of her co-authors is making trouble. All the trouble is of my making (and, more to the point, the Royal Society’s). Someone really has to shine a light on this misbehaviour.

Update (12 March 2014)

I should have noted this before, but on 10 May 2013, the Royal Society sent me an update, explaining some improvements in their process. But as noted in my write-up, it doesn’t actually solve the problem. Doing so would simply require giving three dates: Received, Revised and Accepted. But as I write this, new Proc. B articles still only show Received and Accepted dates.

Reference

Subsequent posts discuss how this issue is developing:

Here’s a blast from the past:

This alleged Compsognathus is a card from the “Flesh” card-game that was printed across several progs (issues) of the comic 2000 AD in 1977. This one is from the back cover of Prog 9. (Click through the picture for the whole back cover.)

“Flesh” was one of the half-dozen or so stories that appeared each week in those early months of 2000 AD. It was the story of how cowboys of the future travelled back to the Mesozoic to harvest dinosaurs for their meat, and was the subject of Jeff Liston’s chapter in the recentish Geological Society volume on the history of dinosaur research.

Compsognathus made another pop-culture appearance in The Lost World: Jurassic Park, of course, as the cute little “compys” that tear one of the nastier human characters to pieces.

Why does the 2000 AD Compsognathus have actinopterygian-like fins for arms? According to Wikipedia, The idea comes from Bidar et al. (1972), who supposed that the French specimen had webbed forefeet, which would look like flippers in life — an idea illustrated as part of a larger scene by Halstead (1975):

John Ostrom’s (1978) Compsognathus monograph showed that this was nonsense, but of course that was too late for the early issues of 2000 AD.

References

Bidar, A.; Demay L., Thomel G. 1972. Compsognathus corallestris, une nouvelle espèce de dinosaurien théropode du Portlandien de Canjuers (Sud-Est de la France). Annales du Muséum d’Histoire Naturelle de Nice 1:9–40.

Halstead L.B. 1975. The evolution and ecology of the dinosaurs. Eurobook. ISBN 0-85654-018-8.

Ostrom, J.H. 1978. The osteology of Compsognathus longipesZitteliana 4:73–118.

Update 1 (the next day)

In a comment below, Andrea Cau points to this post on his blog Theropoda (“the most inclusive blog containing Allosaurus fragilis but not Saltasaurus loricatus) which contains two more flippered-Compsognathus illustrations. Here they are: one from David Lambert’s book Dinosaur! …

… and one from David Norman’s Illustrated Encyclopedia of Dinosaurs.

Update 2 (two days later)

Silly me, I should of course have posted Bidar et al.’s (1972) own life restoration of Compsognathus. It’s not great art, but it’s … actually, I’m not sure what it is. But anyway, here it is:

Attempted reconstruction of attitudes of Compsognathus corallestris nov. sp. A, erect stance (walking); B, sitting (inspired by O. Abel); C, Swimming; D, Diving. (Bidar et al. 1972:figure 21)