The only winning move is not to play
March 19, 2013
The problem
I find myself reading a lot recently about “portable peer-review” — posts like Take me as I am, and my paper as it is? by scicurious at Neurotic Physiology, which excellently diagnoses a terrible, wasteful problem in scientific publishing:
My papers don’t often get in with minor revisions. Often I’ve got a ridiculously puffed head about my own work (apparently), and send them to places which reject them out of hand, or suggest major revisions and piles of new experiments which we just cannot do for various reasons. Then the paper ends up shuttled around. Send it in, wait 3 months, get rejected. Reformat (+2 mo or even more depending on collaborators and how much other crap you’ve got on your plate at the time) and send it out again. Years go by. In the meantime, suggested reviewers begin to hate me and I run out of new ones (only so many people in the field!).
I really wish there was a way to get out of this. This sort of thing contributes to the long lag times and slowness of scientific advance.
What a waste! What a drag on the progress of science! What a ridiculous situation we’re got ourselves into, with our chasing-after-prestigious-journals games.
An inadequate solution
The solution proposed by scicurious is:
You submit a paper to a large umbrella of journals of several “tiers”. It goes out for review. The reviewers make their criticisms. Then they say “this paper is fine, but it’s not impactful enough for journal X unless major experiments A, B, and C are done. However, it could fit into journal Y with only experiment A, or into journal Z with only minor revisions”.
As an incremental improvement on the current system, this is good, if rather impractical to implement.
But it doesn’t go nearly far enough. It still wastes time by going to multiple journals, probably with different formatting requirements, requiring assessment (albeit more lightweight) by several editors. And it does all that in the name of getting a designer label onto the paper by placing it in a “good” journal.
What are we, fourteen?
High-school kids are dumb enough to judge other kids by how fashionable their clothes are, by the labels on them, by whether they’re the clothes other kids think are cool.
Have we really not got beyond that?
The ugly truth
Trying to get into “good” journals is an idiot game. (Notice I don’t say “an idiot’s game” — more on this distinction below.) Although the political and bean-counting value of getting into Nature is huge, the scientific value of getting into Nature is zero. A paper in Nature is literally no better at all than the same paper would be in PLOS ONE. (In fact, it’s probably less good, because it will be butchered to fit the draconian space requirements.) Spending time and effort in trying to get a given piece of research into Nature is just about the least useful thing that can be done for that research.
I think deep down everyone knows this. But of course scientists still waste innumerable hours formatting their work first for Nature, then for Science when it gets rejected, then for PNAS when it gets rejected again, and so on “down the ladder”. But that direction is only “down” by agreement. And the reason of course is because it’s widely (though not universally) believed that wearing these designer clothes is the way to get jobs and grants. That’s why people who are not idiots play this idiot game.
(Thanks heavens for funders and assessors who explicitly state that the journal a work is published in has no effect on how it’s evaluated. You can find such statements from The Wellcome Trust, and regarding the Research Excellence Framework (REF). I want to see more granting and evaluation bodies make similar statements, and I look forward to seeing a university hiring policy that says the same.)
A better way
Happily for me, I don’t need a job or a grant, so I have the luxury of standing on the sidelines, shaking my head sagely yet smugly at the ridiculous manouevres happening on the pitch.
I admit to my shame that I have played the getting-into-a-good-journal game in the past, just because I blindly copied what I saw my colleagues doing without really thinking about it. One result is that our neck-anatomy paper was needlessly held up for more than four years. No-one benefits from these delays. They are a completely avoidable net loss for science.
No more. I am done with having my work rejected for spurious (i.e. non-scientific reasons). I’m only planning on submitting to journals that don’t do that. I reject the idiot notion that the natural lifecycle of a piece of work involves multiple submissions-review-reject cycles. From now on, my cycle is: do some work, write it up, submit it, see it published, move on to the next thing.
And note that “move on to the next thing” is a crucial step here. What really burns me is not the four-year delays on the papers I mentioned above, but all the other work that I’ve not done because I’ve been buggering about, excuse my French, with the corpses of these long-dead projects instead of getting the next thing done. And if that’s true for me, I bet it’s true for you, as well. Yes, you, reading this!
As of now, except in exceptional circumstances, my plan is only to submit to venues where I know scientifically sound work will be accepted. That means “megajournals” like PLOS ONE, PeerJ and (I don’t know, I will look into it) maybe some or all BMC journals. It also means edited volumes that I’m invited to contribute to (though they have their own issues). It probably also means certain other journals, such as PalArch, though they don’t make it explicit (and it would be good if they did).
Three clarifications
First clarification: to be clear, I am not arrogant enough to think this means I will never again have a paper rejected. No doubt there will be occasions where I’ve made significant scientific errors, and reviewers will have to point those out and recommend rejection. I don’t mind that: it’s peer-review actually doing its job, and I’d rather fix those mistakes before publication. What I’m done with is rejections on the basis of “not impacty enough for this journal”, or the often equally specious “not a good fit”.
Second clarification: I don’t absolutely rule out exceptions. There might be occasions where, say, an impact-selective journal announces plans to put out a special volume that I want to be part of. I might submit to that; then again, I might not. I’ll judge it as it comes. But the point is, any exceptions will be exceptions. When I start thinking “where shall I send this?”, my list won’t start with Palaeontology and JVP. I’m glad to have got those notches on my bedpost, but I don’t feel any great need to go back to them.
Third clarification: I do understand that others might not be in a position to make the same leap. I am 99.7% certain that Darren won’t, for example, as he is convinced of the absolute necessity of Science‘n’Nature papers to advance his career. Matt, on the other hand, can and I think will — he’s got a tenure-track job at a university that he likes, has no plans to move on, and doesn’t need “prestigious” papers for his tenure case, only good ones.
(It pained me to have to make that distinction. What a stupid world, where “prestigious papers” and “good papers” are not synonymous, and don’t even overlap that much.)
But for people who, like me, don’t need to have an eye on the possible job-power of “prestige”, it seems obviously better to do what advances science best and fastest. And what a tragedy that advancing science isn’t what gets jobs.
Sauropod Vertebra Picture of the Week 
March 19, 2013 at 11:35 am
Neat article as usual, Mike, and I agree on all counts with the 0 overlap between quality and label. It goes back to the points made on one of your earlier posts, however, that in the current academic system, we are (especially younger researchers) judged by our peers on which journals we publish articles in, irrespective of the wealth of evidence that this is a meaningless measure of quality. Until that changes, most trying to climb up the academic ladder won’t be able to adopt this, arguably much more logical, stance. Pity eh.
March 19, 2013 at 11:52 am
Well, this is where funders, evaluators and job-search committees have so much power to change things. It’s why I am mightily impressed by Wellcome, and by (this aspect of) the REF.
In fact, it would be great to put together a list of funders and evaluators who similarly make an explicit statement that they don’t care what journal work is in, but about the quality of the work. Does anyone know of others?
March 19, 2013 at 11:36 pm
I happen to agree, and I happen to be really glad I’m no longer in academia where I would have had to play that game.
The really scary part (for academicians and others) is that it’s possible for any group of like-minded people to throw together a journal, review articles, and publish online for fairly little money. It would cost rather more to archive stuff in perpetuity, but the point is that the current journals are both expensive and highly politicized entirely by custom and history, not by need. Academic publishing uses the same model as vanity presses, and it appears to be mostly by, for, and about vanity, to the profit of the publishers and the impoverishment of academic scientists (and the public whose taxes support their grants) who pay to play and pay to read the results.
The odd thing is that the rest of the publishing world apparently expected academic publishing to lead the way down the steep slope of digital media plowed by music and videos. Instead, academic publishing has been amazingly reactionary in clinging to the vanity publishing model. Now, in an age when ebooks are killing publishers left and right and anyone can publish their scientific paper on Amazon and sell it for a penny, scientific publishing remains this, well, game.
March 20, 2013 at 6:58 am
Hello, my name is Marius Buliga and I used to be a serial submitter.
March 20, 2013 at 7:43 am
[…] PS2. See also the very interesting post by Mike Taylor “The only winning move is not to play“. […]
July 2, 2013 at 1:00 pm
An interesting initiative I would like to mention in this context is Peerage of Science http://www.peerageofscience.org/, which turns the world of peer review upside down: you submit your paper not to a specific journal, but to be reviewed by your peers, double-blind. Members in your field are notified when your manuscript is submitted, and they can choose themselves to review or not. The deadlines are strict, but a reviewer who initially signed up to do a review (here called “essay”) can always opt out before the deadline. After the deadline, the reviewers are invited to judge each other’s essays. The author then receive the essays and their quality scores, and can submit a revised version (or retract the manuscript), after which the final score is determined for the paper by the original reviewers, weighed by their essay quality score. So as a member, you have an article quality score (for the papers you submitted), an essay quality score (for the reviews you wrote) and a review balance, which prevents peers from submitting too many papers without reviewing any.
Journals subscribing to Peerage of Science can track papers throughout the process and make an offer for publication at any time, until a participating journal has acquired it or the authors have taken it to a journal outside Peerage of Science, where they can also offer access to the reviews in Peerage.
There are now over 1300 peers in this system, mainly from the fields of Ecology, Evolutionary Biology and Conservation Biology, but numbers keep growing. You can become a peer after having published at least one paper in a peer-reviewed journal. It’s invitation only, so you will have to find someone willing to invite you.
April 24, 2015 at 8:04 am
[…] But I hear you on the message sent by acquiescing to a $1600-APC journal, even if that APC is waived. We both want to shift towards a world where there are no journals that charge that kind of money — or at least, that if they do, it’s because they’re the kind of “selective” journal that thinks there’s something praiseworthy about rejecting most scientifically sound submissions. Journals of that kind don’t concern me one way or another, because I just don’t play that game. […]
September 15, 2015 at 8:25 am
[…] was my first encounter with “not significant enough for our journal” — a game that I no longer play. It was to be very far from my last experience of Wrong Mike Taylor […]
August 31, 2016 at 8:55 pm
[…] peer-review. The special sauce that persuaded the US Patent Office that this is a new invention is cascading peer review — an idea so obvious and so well-established that even The Scholarly Kitchen was writing […]
January 12, 2017 at 7:45 pm
Sadly, for those of us just starting in research we simply have no choice but to play the idiot game. I am currently, on a PhD and my supervisor (ego level is incredible) ALWAYS plays the game.
Result? Reformatting and resubmitting ad infinitum this is AFTER every sentence in my drafts have been needlessly rephrased multiple times by my supervisor before we even start submitting.
I am seriously considering whether I can stay with the team after the programme given that I don’t agree with this manner of thinking among others.
But then the research fellowships are not exactly plentiful so when the time comes I wonder if I would be able to leave the team. In fact I might just be glad for a postdoc with them or perhaps go into industry.
February 12, 2017 at 4:44 pm
This is all well and good, but what about the readers? These strange creatures are nowhere to be seen in this discussion. You can publish in PLOS but who will know about it? Are there any people who regularly browse issues of a mega-journal? You say: “Although the political and bean-counting value of getting into Nature is huge, the scientific value of getting into Nature is zero.”
I am not convinced. I think a paper in Nature or, more realistically, in a regular technical journal, will give your article quite an exposure, while your paper in PLOS may not be noticed by anyone – unless you make a very special effort to publicise it among your colleagues. If you have a special strategy, please let us know.
I haven’t heard until today about mega-journals and I have never came across a paper in one. Maybe it is popular in some areas of science but certainly not in my area.
My ignorant opinion is that while criteria of significance and impact are subjective, in mainstream journals they do filter out trivial findings. In contrast, a mega journal important papers will be crowded out by a myriad of trivial and useless results. We certainly publish lots of junk in regular journals but I suspect the proportion should be much worse in mega journals.
Any thoughts?
February 12, 2017 at 5:18 pm
Boris,
I am convinced that the exact opposite is the case. When articles are published in open-access journals, no would-be reader is prevented from reading them. They get read or not according to whether they interest people; they get downloaded or not according to whether they’re good. Which is why our 2013 paper in PeerJ, at that time a completely unknown journal, has been viewed 24,000 times and downloaded 3,000 times. (Also cited 17 times (31 times according to Google Scholar).
Our special strategy for getting our work noticed? Only the obvious things: we blog about out work, sometimes tweet. I think that’s it. We live now in an ecosystem where word-of-mouth is much more important that browsing journal ToCs — a thing I have not done in literally years.
February 13, 2017 at 5:53 am
Well, I did a quick keyword search in PLOS ONE and found no papers remotely related to my research area (very broadly construed). As you specified on another thread, if you are not already familiar with a journal — because it’s published research you respect, or colleagues who you respect have published in it or are on the editorial board — then do not submit your work to that journal. Thus, using these criteria, I should not be submitting my work to this journal or any other mega journal for that matter any time soon.
But you also say “Thanks heavens for funders and assessors who explicitly state that the journal a work is published in has no effect on how it’s evaluated. ” But how a grant assessor is supposed to assess the investigator’s record? Read their papers? That is impractical because of time it takes but also because I often get to assess grants outside my immediate areas of expertise, and thus won’t be able to assess the importance and originality of the findings, even if I understand the logic of the paper. Now I do not give a huge wait to publications in Nature, but record of publication in industry-standard journals is a kind of pre-requisite. Maybe I have to change my habits in the rapidly changing world but I need to find an alternative assessment strategy. And BTW before you ask, the proposal itself is about 40% of the assessment, but another 40% is the quality/research record of investigators, most of which is publications plus self-praise by the applicants.
February 13, 2017 at 5:55 am
obviously weight, not wait :)
February 13, 2017 at 8:43 am
Well, Boris, I don’t know what your research area is. If you’re a geologist or chemist then, sure, you won’t find much in PLOS ONE. If you’re in biomed, you’ll get better results. And, yes, if colleagues in your field are not publishing there, then it’s likely not a good venue for your own work.
But again, searching for papers within a journal — traditional or megajournal — seems like a weird thing to do in 2017. There are so many way to search the whole body of literature. Why would I ever ask “what papers have been published on sauropod neuroendocasts in JVP?” when I could ask “what papers have been published on sauropod neuroendocasts?”
Yes!
This can’t be said too often or too clearly: if assessors (whether for grants, jobs, promotions or tenure) rely on publication venue instead of reading the papers they will reach the wrong conclusion.