Good experiences of peer-review at Qeios

March 9, 2021

A month after I and Matt published our paper “Why is vertebral pneumaticity in sauropod dinosaurs so variable?” at Qeios, we were bemoaning how difficult it was to get anyone to review it. But what a difference the last nineteen days have made!

In that time, we’ve had five reviews, and posted three revisions: revision 2 in response to a review by Mark McMenamin, version 3 in response to a review by Ferdinand Novas, and version 4 in response to reviews by Leonardo Cotts, by Alberto Collareta, and by Eduardo Jiménez-Hidalgo.

Taylor and Wedel (2021: Figure 2). Proximal tail skeleton (first 13 caudal vertebrate) of LACM Herpetology 166483, a juvenile specimen of the false gharial Tomistoma schlegelii. A: close-up of caudal vertebrae 4–6 in right lateral view, red circles highlighting vascular foramina: none in Ca4, two in Ca5 and one in Ca6. B: right lateral view. C: left lateral view (reversed). D: close-up of caudal vertebrae 4–6 in left lateral view (reversed), red circles highlighting vascular foramina: one each in Ca4, Ca5 and Ca6. In right lateral view, vascular foramina are apparent in the centra of caudal vertebrae 5–7 and 9–11; they are absent or too small to make out in vertebrae 1–4, 8 and 12–13. In left lateral view (reversed), vascular foramina are apparent in the centra of caudal vertebrae 4–7 and 9; they are absent or too small to make out in vertebrae 1–3, 8, and 10–13. Caudal centra 5–7 and 9 are therefore vascularised from both sides; 4 and 10–11 from one side only; and 1–3, 8 and 12–13 not at all.

There are a few things to say about this.

First, this is now among our most reviewed papers. Thinking back across all my publications, most have been reviewed by two people; the original Xenoposeidon description was reviewed by three; the same was true of my reassessment of Xenoposeidon as a rebbachisaur, and there may have been one or two more that escape me at the moment. But I definitely can’t think of any papers that have been under five sets of eyes apart from this one in Qeios.

Now I am not at all saying that all five of the reviews on this paper are as comprehensive and detailed as a typical solicited peer review at a traditional journal. Some of them have detailed observations; others are much more cursory. But they all have things to say — which I will return to in my third point.

Second, Qeios has further decoupled the functions of peer review. Traditional peer review combines three rather separate functions: A, Checking that the science is sound before publishing it; B, assessing whether it’s a good fit for the journal (often meaning whether it’s sexy enough); and C, helping the authors to improve the work. When PLOS ONE introduced correctness-only peer-review, they discarded B entirely, reasoning correctly that no-one knows which papers will prove influential[1]. Qeios goes further by also inverting A. By publishing before the peer reviews are in (or indeed solicited), it takes away the gatekeeper role of the reviewers, leaving them with only function C, helping the authors to improve the work. Which means it’s no surprise that …

Third, all five reviews have been constructive. As Matt has written elsewhere, “There’s no way to sugar-coat this: getting reviews back usually feels like getting kicked in the gut”. This is true, and we both have a disgraceful record of allowing harshly-reviewed projects to sit fallow for far too long before doing the hard work of addressing the points made by the reviewers and resubmitting[2].

The contrast with the reviews from Qeios has been striking. Each one has sent me scampering back to the manuscript, keen to make (most of) the suggested changes — hence the three revised versions that I’ve posted in the last fortnight. I think there are at least two reasons for this, a big one and a small one.

  • The big reason, I think, is that the reviewers know their only role is to improve the paper. Well, that’s not quite true: they also have some influence over its evaluation, both in what they write and in assigning a 1-to-5 star score. But they know when they’re writing their reviews that whatever happens, they won’t block publication. This means, firstly, that there is no point in their writing something like “This paper should not be published until the authors do X”; but equally importantly, I think it puts reviewers in a different and more constructive mindset. They feel themselves to be allies of the authors rather than (as can happen) adversaries.
  • The smaller reason is it’s easier to deal with one review at a time. I understand why journals solicit multiple reviews: so the handling editor can consider them all in reaching a decision. I understand why the authors get all the reviews back at once. But that process can’t help but be discouraging: because, once the decision has been made, they’re all on hand and there’s no point in stringing them out. One at a time may not be better, exactly; but it’s emotionally easier.

Is this all upside? Well, it’s too early to say. We’ve only done this once. The experience has certainly been more pleasant — and, crucially, much more efficient — than the traditional publishing lifecycle. But I’m aware of at least two potential drawbacks:

First, the publish-first lifecycle could be exploited by cranks. If the willingness to undergo peer-review is the mark of seriousness in a researcher — and if non-serious researchers are unwilling to face that gauntlet — then a venue that lets you make an end-run around peer-review is an obvious loophole. How serious a danger is this? Only time will tell, but I am inclined to think maybe not too serious. Bad papers on a site like Qeios will attract negative reviews and low scores, especially if they start to get noticed in the mainsteam media. They won’t be seen as having the stamp of having passed peer-review; rather, they will be branded with having publicly failed peer-review.

Second, it’s still not clear where reviewers will come from. We wrote about this problem in some detail last month, and although it’s worked out really well for our present paper, that’s no guarantee that it will always work out this well. We know that Qeios itself approached at least one reviewer to solicit their comments: that’s great, and if they can keep doing this then it will certainly help. But it probably won’t scale, so either a different reviewing culture will need to develop, or we will need people who — perhaps only on an informal basis — take it on themselves to solicit reviews from others. We’re interested to see how this develops.

Anyway, Matt and I have found our first Qeios experience really positive. We’ve come out of it with what I think is a good paper, relatively painlessly, and with much less friction than the usual process. I hope that some of you will try it, too. To help get the process rolling, I personally undertake to review any Qeios article posted by an SV-POW! reader. Just leave a comment here to let me know about your article when it’s up.

 

Notes

[1] “No-one knows which papers will prove influential”. As purely anecdotal evidence for this claim: when I wrote “Sauropod dinosaur research: a historical review” for the Geological Society volume Dinosaurs: A Historical Perspective, I thought it might become a citation monster. It’s done OK, but only OK. Conversely, it never occurred to me that “Head and neck posture in sauropod dinosaurs inferred from extant animals” would be of more than specialist interest, but it’s turned out to be my most cited paper. I bet most researchers can tell similar stories.

[2] One example: my 2015 preprint on the incompleteness of sauropod necks was submitted for publication in October 2015, and the reviews[3] came back that same month. Five and a half years later, I am only now working on the revision and resubmission. If you want other examples, we got ’em. I am not proud of this.

[3] I referred above to “harsh reviews” but in fact the reviews for this paper were not harsh; they were hard, but 100% fair, and I found myself agreeing with about 90% of the criticisms. That has certainly not been true of all the reviews I have found disheartening!

 

16 Responses to “Good experiences of peer-review at Qeios

  1. Matt Wedel Says:

    I think there is another reason why the reviews have been so constructive, and a big one: all reviews at Qeios are signed, and published. You and I both found in reviewing for PeerJ that just the possibility of having our reviews published made us be more careful and more constructive in our comments–and at PeerJ reviewers still have the option to remain anonymous, and authors have the option to not publish the peer reviews. Qeios has removed both of those impediments.

    And I do see them as impediments. As I wrote here just last month, PeerJ’s options to have reviews be (1) signed and (2) published “were both pretty daring steps at the time, but if I could I’d turn both of those into mandates rather than options. Sunlight is the best disinfectant, and I think almost all of the abuses of the peer review system would evaporate if reviewers had to sign their reviews, and all reviews were published alongside the papers.” Qeios just went and did it, and I think their system is superior for having done so.

    The question of whether all reviews will be signed and published is entirely separate from whether review happens pre- or post-publication, but I agree with you that post-publication reviews are more likely to be constructive, for the reasons that you discussed in the post.

    In an interesting way, post-publication review may lead to better papers not only because the reviews may tend to be more constructive, as they were in our case, but also because authors may be more careful to submit rigorously self-edited and polished work if they know that the v0 initial submission will be publicly available forever. The same incentive to do good work, and only good work, because it will all be in the open, falls on authors and reviewers alike. I didn’t think of that until just now, and I can’t help but see it as a good thing.

  2. Mike Taylor Says:

    All excellent points, which I should have included in the post!

  3. dale m. Says:

    To have signed and published reviews, as Matt stated, puts pressure on both author and reviewer to create a good publishable paper. It also points out whether or not the reviewer has the background in your sub-specialty or not … as well as interest. So, some reviewers’ comments R not necessarily going to line up with other reviews. And one reviewer at a time allows an author to tweak a correction B 4 sending it to the 2nd reviewer (if that’s how it is done).

    Now. I’m going to suggest this but I want U to regard this as simply a flight of fancy (because the science, financially, isn’t quite there yet).

    In the distant future, could U ever see a sub-division of any palaeontological society that would set aside a hypothetical “Supreme Court” of 10-15 chosen and “highly paid full time” reviewers that would occupy said position for, say, 3 years ?!? They would not be able to occupy such a position for at least another 3 years. It won’t take the sting out of reviews but it might be able to streamline them. When I say “highly paid”, it is to compensate these reviewers for the time taken away from their own research and teaching assignments. So the pay has to be quite high!

    More importantly, the society would set the rules of engagement within their own science and the occupation of those reviewer chairs would not be appointed outside the science but democratically by the society’s authors. One would have to have published papers in order be eligible to cast such a vote.

    Okay. I’m outta here …

  4. LeeB Says:

    I see the latest version still has the glitch in figure 5.

  5. rkawhite Says:

    Really appreciate you both sharing your experiences here. Mike, your comment in Note 2 “I am not proud of this” made me smile because it exemplified the willingness to openly share your current experience and reflect on past experiences. This is the very impulse behind open research.

    I’ve had the odd researcher ask me about Qeios (I am an open access adviser) and had nothing particularly useful to say — until now, since I can point them in this direction.

  6. Mike Taylor Says:

    @LeeB Arrgh, you’re right! Please leave this as a comment on the paper itself, and I will fix it in the next revision and mention you in the acknowledgements.

  7. LeeB Says:

    Unfortunately it wants me to supply an institutional email address which I don’t have.

  8. Gabri Says:

    Hey @LeeB, you can still sign up with your ORCID (in case un have one) or you can request the creation of a profile by clicking on “Don’t have one?” next to the “Institutional Email Address” field in the sign up page – https://www.qeios.com/register/researcher

  9. Mike Taylor Says:

    That’s bad. Let me see what they can do about this. I started a Twitter thread at https://twitter.com/MikeTaylor/status/1370150176636203017

  10. Mike Taylor Says:

    Oh, I see the Qeios people had replied before I did — but it got held in moderation because it contains links. Does this work for you, LeeB?

  11. LeeB Says:

    I got their email and put a short note there; it actually shows as a review which isn’t exactly what it is but still it is there.

  12. LeeB Says:

    Thanks Gabri but I don’t have an ORCID and when I clicked on the “Don’t have one button” nothing happened; but they contacted me by email and the problem ins now solved.

  13. Matt Wedel Says:

    LeeB, thanks for dealing with all of this faffing about just to help us improve our paper — we’re grateful!

  14. LeeB Says:

    That’s fine I am used to not easily fitting in the world.


  15. […] and BiorXiv (now that it’s moving towards support for reviewing) are the way to go. Our own experience at Qeios has been very good — if it works this well the next time we use it, I think think it’s a keeper. Crucially, I […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: