A funny thing happened on the way to the Shiny Digital Future
February 4, 2021
Picture is unrelated. Seriously. I’m just allergic to posts with no visuals. Stand by for more random brachiosaurs.
Here’s something I’ve been meaning to post for a while, about my changing ideas about scholarly publishing. On one hand, it’s hard to believe now that the Academic Spring was almost a decade ago. On the other, it’s hard for me to accept that PeerJ will be only 8 years old next week–it has loomed so large in my thinking that it feels like it has been around much longer. The very first PeerJ Preprints went up on April 4, 2013, just about a month and a half after the first papers in PeerJ. At that time it felt like things were moving very quickly, and that the landscape of scholarly publishing might be totally different in just a few years. Looking back now, it’s disappointing how little has changed. Oh, sure, there are more OA options now — even more kinds of OA options, and things like PCI Paleo and Qeios feel genuinely envelope-pushing — but the big barrier-based publishers are still dug in like ticks, and very few journals have fled from those publishers to re-establish themselves elsewhere. APCs are ubiquitous now, and mostly unjustified and ruinously expensive. Honestly, the biggest changes in my practice are that I use preprint servers to make my conference talks available, and I use SciHub instead of interlibrary loan.
But I didn’t sit down to write this post so I could grumble about the system like an old hippie. I’ve learned some things in the past few years, about what actually works in scholarly publishing (at least for me), and about my preferences in some areas, which turn out to be not what I expected. I’ll focus on just two areas today, peer review, and preprints.
How I Stopped Worrying and Learned to Love Peer Review
Surprise #1: I’m not totally against peer review. I realize that the way it is implemented in many places is deeply flawed, and that it’s no guarantee of the quality of a paper, but I also recognize its value. This is not where I was 8 years ago; at the time, I was pretty much in agreement with Mike’s post from November, 2012, “Well, that about wraps it up for peer-review”. But then in 2014 I became an academic editor at PeerJ. And as I gained first-hand experience from the other side of the editorial desk, I realized a few things:
- Editors have broad remits in terms of subject areas, and without the benefit of peer reviews by people who specialize in areas other than my own, I’m not fit to handle papers on topics other than Early Cretaceous North American sauropods, skeletal pneumaticity, and human lower extremity anatomy.
- Even at PeerJ, which only judges papers based on scientific soundness, not on perceived importance, it can be hard to tell where the boundary is. I’ve had to reject a few manuscripts at PeerJ, and I would not have felt confident about doing that without the advice of peer reviewers. Even with no perceived importance criterion, there is definitely a lower bound on what counts as a publishable observation. If you find a mammoth toe bone in Nebraska, or a tyrannosaur tooth in Montana, there should probably be something more interesting to say about it, beyond the bare fact of its existence, if it’s going to be the subject of a whole paper.
- In contentious fields, it can be valuable to get a diversity of opinions. And sometimes, frankly, I need to figure out if the author is a loony, or if it’s actually Reviewer #2 that’s off the rails. Although I think PeerJ generally attracts fairly serious authors, a handful of things that get submitted are just garbage. From what I hear, that’s the case at almost every journal. But it’s not always obvious what’s garbage, what’s unexciting but methodologically sound, and what’s seemingly daring but also methodologically sound. Feedback from reviewers helps me make those calls. Bottom line, I do think the community benefits from having pre-publication filters in place.
- Finally, I think editors have a responsibility to help authors improve their work, and reviewers catch a lot of stuff that I would miss. And occasionally I catch something that the reviewers missed. We are collectively smarter and more helpful than any of us would be in isolation, and it’s hard to see that as anything other than a good thing.
The moral here probably boils down to, “white guy stops bloviating about Topic X when he gains actual experience”, which doesn’t look super-flattering for me, but that’s okay.
You may have noticed that my pro-peer-review comments are rather navel-gaze-ly focused on the needs of editors. But who needs editors? Why not chuck the whole system? Set up an outlet called Just Publish Everything, and let fly? My answer is that my time in the editorial trenches has convinced me that such a system will silt up with garbage papers, and as a researcher I already have a hard enough time keeping up with all of the emerging science that I need to. From both perspectives, I want there to be some kind of net to keep out the trash. It doesn’t have to be a tall net, or strung very tight, but I’d rather have something than nothing.
What would I change about peer review? Since it launched, PeerJ has let reviewers either review anonymously, or sign their reviews, and it has let authors decide whether or not to publish the reviews alongside the paper. Those were both pretty daring steps at the time, but if I could I’d turn both of those into mandates rather than options. Sunlight is the best disinfectant, and I think almost all of the abuses of the peer review system would evaporate if reviewers had to sign their reviews, and all reviews were published alongside the papers. There will always be a-holes in the world, and some of them are so pathological that they can’t rein in their bad behavior, but if the system forced them to do the bad stuff in the open, we’d all know who they are and we could avoid them.
Femur of Apatosaurus and right humerus Brachiosaurus altithorax holotype on wooden pedestal (exhibit) with labels and 6 foot ruler for scale, Geology specimen, Field Columbian Museum, 1905. (Photo by Charles Carpenter/Field Museum Library/Getty Images)
Quo Vadis, Preprints?
Maybe the advent of preprints was more drawn out than I know, but to me it felt like preprints went from being Not a Thing, Really, in 2012, to being ubiquitous in 2013. And, I thought at the time, possibly transformative. They felt like something genuinely new, and when Mike and I posted our Barosaurus preprint and got substantive, unsolicited review comments in just a day or two, that was pretty awesome. Which is why I did not expect…
Surprise #2: I don’t have much use for preprints, at least as they were originally intended. When I first confessed this to Mike, in a Gchat, he wrote, “You don’t have a distaste for preprints. You love them.” And if you just looked at the number of preprints I’ve created, you might get that impression. But the vast majority of my preprints are conference talks, and using a preprint server was just the simplest way to the get the abstract and the slide deck up where people could find them. In terms of preprints as early versions of papers that I expect to submit soon, only two really count, neither more recent than 2015. (I’m not counting Mike’s preprint of our vertebral orientation paper from 2019; he’s first author, and I didn’t mind that he posted a preprint, but neither is it something I’d have done if the manuscript was mine alone.)
My thoughts here are almost entirely shaped by what happened with our Barosaurus preprint. We put it up on PeerJ Preprints back in 2013, we got some useful feedback right away, and…we did nothing for a long time. Finally in 2016 we revised the manuscript and got it formally submitted. I think we both expected that since the preprint had already been “reviewed” by commenters, and we’d revised it accordingly, that formal peer review would be very smooth. It was not. And the upshot is that only now, in 2021, are we finally talking about dealing with those reviews and getting the manuscript resubmitted. We haven’t actually done this, mind, we’re just talking about planning to make a start on it. (Non-committal enough for ya?)
Why has it taken us so long to deal with this one paper? We’re certainly capable — the two of us got four papers out in 2013, each of them on a different topic and each of them substantial. So why can’t we climb Mount Barosaurus? I think a big part of it is that we know the world is not waiting for our results, because our results are already out in the world. We’re the only ones being hurt by our inaction — we’re denying ourselves the credit and the respect that go along with having a paper finally and formally published in a peer-reviewed journal. But we can comfort ourselves with the thought that if someone needs our observations to make progress on their own project, we’re not holding them up. Just having the preprint out there has stolen some of our motivation to the get the paper done and out, apparently enough to keep us from doing it at all.
Mike pointed out that according to Google Scholar, our Barosaurus preprint has been cited five times to date, once in its original version and four times in its revised version. But to me, the fact that the Baro manuscript has been cited five times is a fail. Because all of my peer-reviewed papers from 2014-2016, which have been out for less long, have been cited more. So I read that as people not wanting to cite it. And who can blame them? Even I thought it would be supplanted by the formally-published, peer-reviewed paper within a few weeks or months.
Mike then pointed me to his 2015 post, “Four different reasons to post preprints”, and asked how many of those arguments still worked for me now. Number 2 is good, posting material that would otherwise never see the light of day — it’s basically what I did when I put my dissertation on arXiv. Ditto for 4, which is posting conference presentations. I’m not moved by either 1 or 3. Number 3 is getting something out to the community as quickly as possible, just because you want to, and number 1 is getting feedback as quickly as possible. The reason that neither of those move me is that they’re solved to my satisfaction by existing peer-reviewed outlets. I don’t know of any journals that let reviewers take 2-4 months to review a paper anymore. I don’t know how much credit for the acceleration should go to PeerJ, which asks for reviews in 10 to 14 days, but surely some. And I don’t usually have a high enough opinion of my own work to think that the community will suffer if it takes a few months for a paper to come out through the traditional process.
(If it seems like I’m painting Mike as relentlessly pro-preprint, it’s not my intent. Rather, I’d dropped a surprising piece of news on him, and he was strategically probing to determine the contours of my new and unexpected stance. Then I left the conversation to come write this post while the ideas were all fresh in my head. I hope to find out what he thinks about this stuff in the comments, or ideally in a follow-up post.)
Back to task: at least for me, a preprint of a manuscript I’m going to submit anyway is a mechanism to get extra reviews I don’t want*, and to lull myself into feeling like the work is done when it’s not. I don’t anticipate that I will ever again put up a preprint for one of my own manuscripts if there’s a plausible path to traditional publication.
* That sounds awful. To people who have left helpful comments on my preprints: I’m grateful, sincerely. But not so grateful that I want to do the peer review process a second time for zero credit. I didn’t know that when I used to file preprints of manuscripts, but I know it now, and the easiest way for me to not make more work for both of us is to not file preprints of things I’m planning to submit somewhere anyway.
So much for my preprints; what about those of other people? Time for another not-super-flattering confession: I don’t read other people’s preprints. Heck, I don’t have time to keep up with the peer-reviewed literature, and I have always been convinced by Mike’s dictum, “The real value of peer-review is not as a mark of correctness, but of seriousness” (from this 2014 post). If other people want me to part with my precious time to engage with their work, they can darn well get it through peer review. And — boomerang thought — that attitude degrades my respect for my own preprint manuscripts. I wouldn’t pay attention to them if someone else had written them, so I don’t really expect anyone else to pay attention to the ones that I’ve posted. In fact, it’s extremely flattering that they get read and cited at all, because by my own criteria, they don’t deserve it.
I have to stress how surprising I find this conclusion, that I regard my own preprints as useless at best, and simultaneously extra-work-making and motivation-eroding at worst, for me, and insufficiently serious to be worthy of other people’s time, for everyone else. It’s certainly not where I expected to end up in the heady days of 2013. But back then I had opinions, and now I have experience, and that has made all the difference.
The comment thread is open. What do you think? Better still, what’s your experience?
Sauropod Vertebra Picture of the Week 
February 4, 2021 at 4:56 pm
“If you find a mammoth toe bone in Nebraska, or a tyrannosaur tooth in Montana, there should probably be something more interesting to say about it, beyond the bare fact of its existence, if it’s going to be the subject of a whole paper.”
I am not 100% convinced that’s right. I reviewed a paper a few years ago which basically just said Hey, we found an Elasmosaurus cervical in a drawer. That’s in PLOS ONE, and I think rightly so.
(I reviewed it by accident, BTW: they thought they were getting the other Mike Taylor, the one who specialises in marine reptiles. Instead, they got the Mike Taylor who specialises in necks. But no damage done, since the paper was about a marine reptile neck.)
February 4, 2021 at 5:03 pm
Dude, you are making my point for me. The actual title of the linked paper is, “Revised Vertebral Count in the “Longest-Necked Vertebrate” Elasmosaurus platyurus Cope 1868, and Clarification of the Cervical-Dorsal Transition in Plesiosauria”, which means the authors found at least two more interesting things to say, beyond the bare fact of having found a vertebra in a drawer.
February 4, 2021 at 11:08 pm
I see your points as valid but pretty skewed (by your own experience, which is inevitable). I will not try to modify your conclusions within your scenario, but instead mention some points that may expose its limited range. Picture a (more common?) scenario with some critical differences:
(1) Peer-review in PeerJ (and some others) is anything but standard in the publication-system-as-intended. The role of an academic editor in that environment (e.g., to help the author; to always probe for the minimum tolerable) is VERY different to one managed to filter for predicted impact and the future measurements of success that will come with it for the journal [including the prestige that comes with “rejecting a lot” ¿really a journal publishes rejecting rates as a success measurement? it should be interpreted as a failure of the editorial task or the audience it is attracting!). Back to peer-review, do peer-reviewers help the editor? (Not considering particular experiences,) Yes. However, that says nothing for which of the contrasting editorial/managerial goals they are helping (or even for which of them they are asked, explicitly or not, to contribute with).
(2) You say you use SciHub to get access to papers you want to read. Maybe (I don’t know) you would have access anyway but value speed. Maybe it’s your only reasonable option to get to read. Put yourself in the second scenario and value preprints (or pre-reviewed versions in a green-server) then. As individual items (your point) or as a system that limits (a bit) how much artificial scarcity can be imposed [And I think I am not proposing a dystopic scenario; not-SciHub can happen tomorrow, or at least that is the intention of some actors in the system that already proved powerful in many, many ways].
So, do I prefer a whole publishing system that mimics your experienced conditions? Except for restrictive APCs accepted as a natural choice for OA, yes indeed. Add academic recognition for peer-reviewers and unpaid editors (they are helping the collective under goal N°1!) and an editorial mission of publishing the best version of what was submitted (and shine among prospective authors for that service: “I always send my MSs to JournalX because they really help me to communicate clearly and beautiful”).
ONLY THEN I will help you convince that old hippie of yours ;)
February 5, 2021 at 1:57 am
Coming from a perspective outside academia, I think that one of the most important roles peer review plays is that it helps people who aren’t familiar with the nuances of the field to develop an initial impression of whether something is credible science or the ramblings of a crank, which is valuable given how easily misinformation spreads on the Internet. It isn’t a perfect proxy for scientific quality, but it’s better than nothing in my opinion.
February 5, 2021 at 7:25 am
John, I agree with all of that. Here’s a relevant quote from Mike’s 2014 post on the value of peer review:
“I would imagine that everyone involved in dinosaur research occasionally gets unsolicited emails from cranks and from as-yet unpublished amateurs. One of the most reliable ways to distinguish the two groups is this: serious amateurs are trying to figure out how to get their work into peer-review, while cranks are either actively avoiding it or not even aware of it.”
I have never seen any evidence to contradict that assessment.
February 5, 2021 at 7:27 am
Fair Miles, you raise a lot of good points, particularly regarding the navel-gazey-ness of my assessments in the post, and on the value of preprints as a green OA mechanism for circumventing paywalls. I don’t have much to add other than that I agree with you.
February 5, 2021 at 8:08 pm
I’m coming from this from an outsiders perspective as well (I’m a high school science teacher who makes skeletal diagrams for fun). I generally agree with your points on peer-review. Obviously the system isn’t perfect, but it does guarantee the paper has had at least some vetting. It doesn’t make the conclusions valid and or mean the paper is perfect (it’s the year of our Lord 2021 and folks are still writing descriptive papers on interesting or important specimens without a single measurement in the entire paper!), but it at least suggest the person isn’t a complete nutter. I do think rejecting or approving on the basis of “impact” or “importance” isn’t ideal, but your work does need to meet some standards to get the validity associated with publication.
When it comes to peer-review and anonymity I’m more on the fence. I like that PeerJ has the option to make the review process public and I’ve often found those reviews to be an interesting read. I wouldn’t mandate it though. It would make it harder for bad reviewers to get away with things, but it also introduces new issues as well. Would young, less-established researchers feel comfortable reviewing papers from major figures if they knew their name would be attached? My old advisor once got a completely crap figure from a celebrity in the field who was also known for being an ass behind the scenes. Paleo certainly has had its share of those folks as well.
As for pre-prints, I mostly agree again. As someone who really isn’t “in” paleo, there’s often too many unknowns to really trust them. If I find an old preprint that hasn’t been published I’m left wondering what was wrong with it to keep it from getting published, even though there are a number of other reasons the project might have stalled. I referenced your *Barosaurus* preprints when I was working on a few skeletals, but that was exception I made because 1) I’m familiar with your work and 2) the information I was pulling wasn’t particularly controversial (pictures and measurements mostly).
From a research perspective I don’t think publishing papers as preprints has a ton of value. It likely won’t be used much by others in the field and arguably muddies what is viewed as “legitimate” to those who aren’t in the field. It is handy for talks and slides though.
I do think this is an interesting conversation to have and I can’t wait to see the comments (and likely further posts) that come out off this.
February 5, 2021 at 8:34 pm
Would young, less-established researchers feel comfortable reviewing papers from major figures if they knew their name would be attached?
That’s a legit concern, but at least if the review was signed and published, everyone would know about it, so they could be on the lookout for any retaliation from the major figures.
I think the potential for retaliation over critical reviews must push peer review systems either to double-blind peer review, or open peer review. My issue with double-blind peer review is that many areas of science are so specialized that only a handful of people are qualified to review certain papers, and when you work in one of those narrow fields, it doesn’t take long to learn to recognize everyone else’s writing and reviewing style. At which point the reviews are double-blind in name only, because the reviewer probably recognizes the author and vice versa.
Fully open peer review has the advantage that if anyone misbehaves, either in the review process, or as a result of the review process, it will be visible to the entire community.
February 6, 2021 at 4:25 pm
Arguments either way about reviewer anonymity assume that authors are well-behaved and professional. They. Are. Not.
I have had an author call me on the phone and lambaste me — pretty abusively and saying things meant to sting — about a review I absolutely did not write on a paper I didn’t even know existed! (No, Matt, authors cannot reliably identify reviewers.)
After ending that call and blinking a few times in astonishment, I called the author back and asked whether the person who DID write that review had claimed to be me. (!! a possibility for abuse that had not occurred to me before that moment !!) Fortunately that was not the case, although they had used a similar format of review.
My doctoral advisor used that format, taught that format, and had a lot of students over the years.
SO I decided to distinguish myself from my advisor and his other former students by always being a non-anonymous reviewer.
After making this decision I reviewed a paper that needed some improvement, and the editor sent it back to the authors for revision, *and the lead author called me on the phone* to lobby me to accept it without the revisions I felt it needed.
Hey I’m tenured and in a job I like, so I just said “uh huh, mmm, uh huh” to this spate of authorial self-justification by a senior individual and then did whatever I was already going to do. But not everyone is in that comfortable a position.
SO, my conclusion here is that while we tend to demonize the behavior of reviewers, we are in fact in a situation in which reviewers also have to be protected from authors.
I have, over the years, developed some judgment of which authors will act ethically and which will not, and I won’t review unless I have the choice of whether to do so anonymously or identify myself.
February 6, 2021 at 4:58 pm
Well, Anne, you are right that the potential for abuse exists in both directions. But my own experience has been that I have been on the end of what I consider poor behaviour much more as an author than as a reviewer. So without at all denying that there are arguments in both directions, my inclination is still very much that the cloak of anonymity enables more damage than it prevents.
February 6, 2021 at 8:21 pm
(No, Matt, authors cannot reliably identify reviewers.)
Well, I didn’t argue that they always could, only that they could sometimes, so the supposed anonymity of double-blind peer review is prone to breaking down, in ways that the lack of anonymity in open peer review is not. There’s an asymmetry there in what each method promises to do, versus what it actually does–again, sometimes, not always.
But I admit that my analysis didn’t take into account people going outside the system to call or email and harangue someone outside the peer review system. One possible solution would be to have a statement with each published paper and each published peer review about whether the author or reviewer was contacted outside the system, like the author workload breakdown and COI statements most journals these days require.
I won’t review unless I have the choice of whether to do so anonymously or identify myself.
Interesting. And a pretty compelling argument.
What’s your preference on publishing peer reviews alongside papers? My feeling is that it’s a good thing, whose benefits outweigh any deficits. For one thing, it might keep a-hole reviewers from behaving badly even if they choose to review anonymously: if an assassination review is published, the authors might not recognize the reviewer by their style, but someone else might.
But also, I like published reviews because they allow reviewers to get credit for their work, and they allow that work to be visible and citable. I recognize that some parts of the system need to be designed with the goal of protecting people from bad actors foremost–and I think that’s where anonymity vs openness in peer review comes in–but I also think the system should have a way to reward good actors, and allowing good reviewers to get credit for their work (should they choose to accept it) seems like a good way to do that.
February 6, 2021 at 10:15 pm
People are people, and we can all probably recall some particular (awful, funny, crazy, unacceptable, admirable) anecdote. But c’mon guys! If Wikipedia managed to celebrate its 20th birthday (touching every single sensible topic possible) with open, and even anonymous, contributions, editions and revisions, open publishing and open review still does not exist because
(1) our scientific system is (current scientists are) not interested enough, and/or
(2) journals want to hold peer-review as part of their “irreplaceable” services, and/or
(3) publishers want to hold their journals as prestige brands so “selective” as they posibly can.
[oops, sorry, that was the old hippie hacking my account xD]
February 6, 2021 at 10:37 pm
Pretty fair indictment, Fair Miles–er, old hippie hacker.
I think it’s a combination of all 3:
(1) for sure most scientists don’t care enough; look how hard it is to get many of them to even acknowledge that they, their institutions, and society at large are getting taken to the bank by barrier-based publishers;
(2) yes, journals definitely have a vested interested in promulgating the lie that the peer review academics provide for free is actually one of the services that the same academics pay publishers to provide;
(3) to some extent, although I have been alarmed that as new mega-journals with correctness-only peer review have come online, they don’t seem to have shifted the conversation much on peer review, nor has the average APC gone down. If anything, it’s gone up.
February 6, 2021 at 11:15 pm
What’s your preference on publishing peer reviews alongside papers?
More than once in my life when I have been a reviewer on a paper that came back for multiple rounds, I have learned something cool and/or significant by reading another review. Usually this is something about methods. So I see some potential for significant scientific advantages.
On the other hand, it is going to slow down the review process a LOT if every reviewer has to write for publication. This is particularly true of people like me with perfectionist tendencies. It’s one thing to write a review that is detailed, organized, constructive, and deeply-considered. It’s another thing to do that with flawless grammar and spelling while worrying some dino-fanboy is going to come for you because you’ve mis-typed “Futalognkosaurus.” Or God help it, you made the utterly uncontroversial factual statement that mammals eat dinosaurs — ooo, yeah, I did do that last one. Did you ever see the *entire Web page* entitled, “Anne W is stupid” and dedicated to Anne W’s “stupid” “mistakes”? Maybe it’s on the wayback machine.
And then there’s the reviewer who re-runs the analysis. Or who runs another analysis to check whether assumption “x” is ever true. On the one hand, I’d like credit for that. On the other hand, particularly in the second case, I’m happy to tell the authors what I did if they need to check what they assumed, but it also ends up being my original research which I could publish myself. (That’s happened.) In that case it’s a professional disadvantage to have the review become a publication subsidiary to someone else’s.
So again, this should be a choice for both parties. And I would advise against publishing anonymous reviews.
Back to the subject of abuse:
As an author I have not received a gross hack job of a review. As an associate editor I saw a few and opted to get an additional or alternate review and/or dash off some instructions to the author, such as, “ignore points 1,3,and 4; however points 2 and 5 should be addressed in revision.” They aren’t as common as supposed, and editors have options. Editors don’t have to take all the reviewer’s advice.
if an assassination review is published, the authors might not recognize the reviewer by their style, but someone else might.
OR, someone utterly unrelated could be erroneously “recognized,” pilloried on social media, harassed on the telephone, have their Dean called about their supposed review… …What a FREAKING NIGHTMARE. DO NOT encourage people to play the guessing game.
Because you know what? The person erroneously piled on is probably more likely to be junior, more likely to be female, and possibly also more likely to be a person of color. Someone who is expected to be submissive and is juuuuust enough of an outsider that people feel comfortable attacking.
February 7, 2021 at 12:29 am
DO NOT encourage people to play the guessing game.
Of course not! I’ve seen guesses go wrong myself. I know of a case in which a senior person held up a publication by a junior person over a negative review that wasn’t actually written by said junior person. (I wasn’t the senior person, the junior person, or the person who wrote the review, I just happen to know who all of those folks are.) My point is just that double-blind peer review is not as airtight as advertised.
I’m happy to tell the authors what I did if they need to check what they assumed, but it also ends up being my original research which I could publish myself. (That’s happened.) In that case it’s a professional disadvantage to have the review become a publication subsidiary to someone else’s.
Ah, yeah, that’s interesting. On one hand, my utopian tendency says that the solution is to erase the artificial perceptual division between reviews and papers, so that having that analysis in a review does count as having published it yourself. On the other three limbs, we’re not there yet, we may never be, and I think your last point would still be valid, that the reviewer’s contribution would always be seen as a subsidiary.
As for your other points, you’ve given me a lot to think about — thank you.
February 10, 2021 at 4:14 pm
One place where preprints have been detrimental is with the pandemic of 2020. There were a number of alarming events/news stories based on preprints which may or may not have ultimately been published. I would ask my son, who is a molecular biophysicist, about alarming news reports and he would sometimes say it was based on a posted preprint that was being savaged for poor data analysis, poor or biased data collection, too small a sample size, etc. But when news sources are looking for any info during a national health crisis like that, they latch on to any source and made little differentiation between a peer reviewed publication, a posted preprint, or (for that matter) a company’s press release. I suppose it is more a problem of the preprint user and the pressure to be first with the story during a crisis, than the preprint process itself. I soon learned to look at news reports on COVID to see if they mentioned the study was yet to be published, which likely indicates a preprint.
February 10, 2021 at 4:37 pm
Thanks, Dan, you raise a lot of interesting issues.
On one hand, biomedical knowledge has a much more immediate and direct impact on human well-being than, say, noodling about sauropods, so we might want the net to be strung a little tighter there.
On the other hand, there are lots of reasons to put up preprints, and especially in the midst of a global health emergency, some authors might have wanted to get their data posted ASAP so either researchers could start using it right away, without getting bogged down in the traditional publication cycle. Only the medium they used to distribute their data is also a medium that health reporters are now keeping an eye on, so there’s potential for big mismatches between intent and outcome.
On the third hand, with newsrooms throwing everything against the wall I’m sure fact- and source-checking broke down a little bit from normal, and a lot of medical reporting is pretty dismal even under optimal circumstances.
On the fourth hand, preprints are still a fairly new form of science communication and although we are familiar with them in paleo from long exposure, I get the impression that folks in a lot of other fields, especially those outside of science (like, say, medical journalism), are still figuring out what to do with the new toy.
On reflection, maybe my 2nd, 3rd, and 4th points are all fingers on one giant hand? Or maybe I’m just overreaching with this metaphor.
October 3, 2022 at 8:43 am
[…] since then. I’m much less interested in preprints than I used to be, for reasons explained in this post, and I think Mike sees them as rather pointless if you’re writing the paper in the open […]