Picture is unrelated. Seriously. I’m just allergic to posts with no visuals. Stand by for more random brachiosaurs.

Here’s something I’ve been meaning to post for a while, about my changing ideas about scholarly publishing. On one hand, it’s hard to believe now that the Academic Spring was almost a decade ago. On the other, it’s hard for me to accept that PeerJ will be only 8 years old next week–it has loomed so large in my thinking that it feels like it has been around much longer. The very first PeerJ Preprints went up on April 4, 2013, just about a month and a half after the first papers in PeerJ. At that time it felt like things were moving very quickly, and that the landscape of scholarly publishing might be totally different in just a few years. Looking back now, it’s disappointing how little has changed. Oh, sure, there are more OA options now — even more kinds of OA options, and things like PCI Paleo and Qeios feel genuinely envelope-pushing — but the big barrier-based publishers are still dug in like ticks, and very few journals have fled from those publishers to re-establish themselves elsewhere. APCs are ubiquitous now, and mostly unjustified and ruinously expensive. Honestly, the biggest changes in my practice are that I use preprint servers to make my conference talks available, and I use SciHub instead of interlibrary loan.

But I didn’t sit down to write this post so I could grumble about the system like an old hippie. I’ve learned some things in the past few years, about what actually works in scholarly publishing (at least for me), and about my preferences in some areas, which turn out to be not what I expected. I’ll focus on just two areas today, peer review, and preprints.

How I Stopped Worrying and Learned to Love Peer Review

Surprise #1: I’m not totally against peer review. I realize that the way it is implemented in many places is deeply flawed, and that it’s no guarantee of the quality of a paper, but I also recognize its value. This is not where I was 8 years ago; at the time, I was pretty much in agreement with Mike’s post from November, 2012, “Well, that about wraps it up for peer-review”. But then in 2014 I became an academic editor at PeerJ. And as I gained first-hand experience from the other side of the editorial desk, I realized a few things:

• Editors have broad remits in terms of subject areas, and without the benefit of peer reviews by people who specialize in areas other than my own, I’m not fit to handle papers on topics other than Early Cretaceous North American sauropods, skeletal pneumaticity, and human lower extremity anatomy.
• Even at PeerJ, which only judges papers based on scientific soundness, not on perceived importance, it can be hard to tell where the boundary is. I’ve had to reject a few manuscripts at PeerJ, and I would not have felt confident about doing that without the advice of peer reviewers. Even with no perceived importance criterion, there is definitely a lower bound on what counts as a publishable observation. If you find a mammoth toe bone in Nebraska, or a tyrannosaur tooth in Montana, there should probably be something more interesting to say about it, beyond the bare fact of its existence, if it’s going to be the subject of a whole paper.
• In contentious fields, it can be valuable to get a diversity of opinions. And sometimes, frankly, I need to figure out if the author is a loony, or if it’s actually Reviewer #2 that’s off the rails. Although I think PeerJ generally attracts fairly serious authors, a handful of things that get submitted are just garbage. From what I hear, that’s the case at almost every journal. But it’s not always obvious what’s garbage, what’s unexciting but methodologically sound, and what’s seemingly daring but also methodologically sound. Feedback from reviewers helps me make those calls. Bottom line, I do think the community benefits from having pre-publication filters in place.
• Finally, I think editors have a responsibility to help authors improve their work, and reviewers catch a lot of stuff that I would miss. And occasionally I catch something that the reviewers missed. We are collectively smarter and more helpful than any of us would be in isolation, and it’s hard to see that as anything other than a good thing.

The moral here probably boils down to, “white guy stops bloviating about Topic X when he gains actual experience”, which doesn’t look super-flattering for me, but that’s okay.

You may have noticed that my pro-peer-review comments are rather navel-gaze-ly focused on the needs of editors. But who needs editors? Why not chuck the whole system? Set up an outlet called Just Publish Everything, and let fly? My answer is that my time in the editorial trenches has convinced me that such a system will silt up with garbage papers, and as a researcher I already have a hard enough time keeping up with all of the emerging science that I need to. From both perspectives, I want there to be some kind of net to keep out the trash. It doesn’t have to be a tall net, or strung very tight, but I’d rather have something than nothing.

What would I change about peer review? Since it launched, PeerJ has let reviewers either review anonymously, or sign their reviews, and it has let authors decide whether or not to publish the reviews alongside the paper. Those were both pretty daring steps at the time, but if I could I’d turn both of those into mandates rather than options. Sunlight is the best disinfectant, and I think almost all of the abuses of the peer review system would evaporate if reviewers had to sign their reviews, and all reviews were published alongside the papers. There will always be a-holes in the world, and some of them are so pathological that they can’t rein in their bad behavior, but if the system forced them to do the bad stuff in the open, we’d all know who they are and we could avoid them.

Femur of Apatosaurus and right humerus Brachiosaurus altithorax holotype on wooden pedestal (exhibit) with labels and 6 foot ruler for scale, Geology specimen, Field Columbian Museum, 1905. (Photo by Charles Carpenter/Field Museum Library/Getty Images)

Quo Vadis, Preprints?

Maybe the advent of preprints was more drawn out than I know, but to me it felt like preprints went from being Not a Thing, Really, in 2012, to being ubiquitous in 2013. And, I thought at the time, possibly transformative. They felt like something genuinely new, and when Mike and I posted our Barosaurus preprint and got substantive, unsolicited review comments in just a day or two, that was pretty awesome. Which is why I did not expect…

Surprise #2: I don’t have much use for preprints, at least as they were originally intended. When I first confessed this to Mike, in a Gchat, he wrote, “You don’t have a distaste for preprints. You love them.” And if you just looked at the number of preprints I’ve created, you might get that impression. But the vast majority of my preprints are conference talks, and using a preprint server was just the simplest way to the get the abstract and the slide deck up where people could find them. In terms of preprints as early versions of papers that I expect to submit soon, only two really count, neither more recent than 2015. (I’m not counting Mike’s preprint of our vertebral orientation paper from 2019; he’s first author, and I didn’t mind that he posted a preprint, but neither is it something I’d have done if the manuscript was mine alone.)

My thoughts here are almost entirely shaped by what happened with our Barosaurus preprint. We put it up on PeerJ Preprints back in 2013, we got some useful feedback right away, and…we did nothing for a long time. Finally in 2016 we revised the manuscript and got it formally submitted. I think we both expected that since the preprint had already been “reviewed” by commenters, and we’d revised it accordingly, that formal peer review would be very smooth. It was not. And the upshot is that only now, in 2021, are we finally talking about dealing with those reviews and getting the manuscript resubmitted. We haven’t actually done this, mind, we’re just talking about planning to make a start on it. (Non-committal enough for ya?)

Why has it taken us so long to deal with this one paper? We’re certainly capable — the two of us got four papers out in 2013, each of them on a different topic and each of them substantial. So why can’t we climb Mount Barosaurus? I think a big part of it is that we know the world is not waiting for our results, because our results are already out in the world. We’re the only ones being hurt by our inaction — we’re denying ourselves the credit and the respect that go along with having a paper finally and formally published in a peer-reviewed journal. But we can comfort ourselves with the thought that if someone needs our observations to make progress on their own project, we’re not holding them up. Just having the preprint out there has stolen some of our motivation to the get the paper done and out, apparently enough to keep us from doing it at all.

Mike pointed out that according to Google Scholar, our Barosaurus preprint has been cited five times to date, once in its original version and four times in its revised version. But to me, the fact that the Baro manuscript has been cited five times is a fail. Because all of my peer-reviewed papers from 2014-2016, which have been out for less long, have been cited more. So I read that as people not wanting to cite it. And who can blame them? Even I thought it would be supplanted by the formally-published, peer-reviewed paper within a few weeks or months.

Mike then pointed me to his 2015 post, “Four different reasons to post preprints”, and asked how many of those arguments still worked for me now. Number 2 is good, posting material that would otherwise never see the light of day — it’s basically what I did when I put my dissertation on arXiv. Ditto for 4, which is posting conference presentations. I’m not moved by either 1 or 3. Number 3 is getting something out to the community as quickly as possible, just because you want to, and number 1 is getting feedback as quickly as possible. The reason that neither of those move me is that they’re solved to my satisfaction by existing peer-reviewed outlets. I don’t know of any journals that let reviewers take 2-4 months to review a paper anymore. I don’t know how much credit for the acceleration should go to PeerJ, which asks for reviews in 10 to 14 days, but surely some. And I don’t usually have a high enough opinion of my own work to think that the community will suffer if it takes a few months for a paper to come out through the traditional process.

(If it seems like I’m painting Mike as relentlessly pro-preprint, it’s not my intent. Rather, I’d dropped a surprising piece of news on him, and he was strategically probing to determine the contours of my new and unexpected stance. Then I left the conversation to come write this post while the ideas were all fresh in my head. I hope to find out what he thinks about this stuff in the comments, or ideally in a follow-up post.)

Back to task: at least for me, a preprint of a manuscript I’m going to submit anyway is a mechanism to get extra reviews I don’t want*, and to lull myself into feeling like the work is done when it’s not. I don’t anticipate that I will ever again put up a preprint for one of my own manuscripts if there’s a plausible path to traditional publication.

* That sounds awful. To people who have left helpful comments on my preprints: I’m grateful, sincerely. But not so grateful that I want to do the peer review process a second time for zero credit. I didn’t know that when I used to file preprints of manuscripts, but I know it now, and the easiest way for me to not make more work for both of us is to not file preprints of things I’m planning to submit somewhere anyway.

So much for my preprints; what about those of other people? Time for another not-super-flattering confession: I don’t read other people’s preprints. Heck, I don’t have time to keep up with the peer-reviewed literature, and I have always been convinced by Mike’s dictum, “The real value of peer-review is not as a mark of correctness, but of seriousness” (from this 2014 post). If other people want me to part with my precious time to engage with their work, they can darn well get it through peer review. And — boomerang thought — that attitude degrades my respect for my own preprint manuscripts. I wouldn’t pay attention to them if someone else had written them, so I don’t really expect anyone else to pay attention to the ones that I’ve posted. In fact, it’s extremely flattering that they get read and cited at all, because by my own criteria, they don’t deserve it.

I have to stress how surprising I find this conclusion, that I regard my own preprints as useless at best, and simultaneously extra-work-making and motivation-eroding at worst, for me, and insufficiently serious to be worthy of other people’s time, for everyone else. It’s certainly not where I expected to end up in the heady days of 2013. But back then I had opinions, and now I have experience, and that has made all the difference.

The comment thread is open. What do you think? Better still, what’s your experience?