I’ll try to live-blog the first day of part 2 of the Royal Society’s Future of Scholarly Scientific Communication meeting, as I did for the first day of part 1. We’ll see how it goes.

Here’s the schedule for today and tomorrow.

Session 1: the reproducibility problem

Chair: Alex Halliday, vice-president of the Royal Society

Introduction to reproducibility. What it means, how to achieve it, what role funding organisations and publishers might play.

For an introduction/overview, see #FSSC – The role of openness and publishers in reproducible research.

Michele Dougherty, planetary scientist

It’s very humbling being at this meeting, when it’s so full of people who have done astonishing things. For example, Dougherty discovered an atmosphere around one of Saturn’s moons by an innovative use of magnetic field data. So many awesome people.

Her work is largely to do with very long-term project involving planetary probes, e.g. the Cassini-Huygens probe. It’s going to be interesting to know what can be said about reproducibility of experiments that take decades and cost billions.

“The best science output you can obtain is as a result of collaboration with lots of different teams.”

Application of reproducibility here is about making the data from the probes available to the scientific community — and the general public — so that the result of analysis can be reproduced. So not experimental replication.

Such data often has a proprietary period (essentially an embargo) before its public release, partly because it’s taken 20 years to obtain and the team that did this should get the first crack at it. But it all has to be made publicly available.

Dorothy Bishop, chair of Academy of Medical Sciences group on replicability

The Royal Society is very much not the first to be talking about replicability — these discussions have been going on for years.

About 50% of studies in Bishop’s field are capable of replication. Numbers are even worse in some fields. Replication of drug trials are particularly important, as false result kill people.

Journals cause awful problems with impact-chasing: e.g. high-impact journals will publish sexy-looking autism studies with tiny samples, which no reputable medical journal would publish.

Statistical illiteracy is very widespread. Authors can give the impression of being statistically aware but in a superficial way.

Too much HARKing going on (Hypothesising After Results Known — searching a dataset for anything that looks statistically significant in the shallow p < 0.05 sense.)

“It’s just assumed that people doing research, know what they are doing. Often that’s just not the case.”

many more criticisms of how the journal system encourages bad research. They’re coming much faster than I can type them. This is a storming talk, I wish the record would be made available.

Employers are also to blame for prioritising expensive research proposals (= large grants) over good ones.

All of this causes non-replicable science.

Floor discussion

Lots of great stuff here that I just can’t capture, sorry. Best follow the tweet stream for the fast-moving stuff.

One highlight: Pat Brown thinks it’s not necessarily a problem if lots of statistically underpowered studies are performed, so long as they’re recognised as such. Dorothy Bishop politely but emphatically disagrees: they waste resources, and produce results that are not merely useless but actively wrong and harmful.

David Colhoun comments from the floor: while physical sciences consider “significant results” to be five sigmas (p < 0.000001), biomed is satisfied with slightly less than two sigmas (p < 0.05) which really should be interpreted only as “worth another look”.

Dorothy Bishop on publishing data, and authors’ reluctance to do so: “It should be accepted as a cultural norm that mistakes in data do happen, rather than shaming people who make data open.”

Coffee break

Nothing to report :-)

Session 2: what can be done to improve reproducibility?

Iain Hrynaszkiewicz, head of data, Nature

In an analysis of retractions of papers in PubMed Central, 2/3 were due to fraud and 20% due to error.

Access to methods and data is a prerequisite for replicability.

Pre-registration, sharing of data, reporting guidelines all help.

“Open access is important, but it’s only part of the solution. Openness is a means to an end.”

Hrynaszkiewicz says text-miners are a small minority of researchers. [That is true now, but I and others are confident this will change rapidly as the legal and technical barriers are removed: it has to, since automated reading is the only real solution to the problem of keeping up with an exponentially growing literature. — Ed.]

Floor discussion

When a paper goes for peer-review at PLOS ONE, the reviewers are told not to make any judgement about how important or sexy or “impacty” the paper is — to judge it only on methodical soundness. All papers that are judged sound are to be published without making guesses about which will and won’t improve the journal’s reputation through being influential down the line. (Such guesses are hopelessly inaccurate anyway.)

When PLOS ONE was new, this approach drew scorn from established publishers, but now those publishers all have their own journals that use similar editorial criteria (Nature’s Scientific Reports, AAAS‘s Science Advances, Elsevier’s first attempt, Elsevier’s second attempt, the Royal Society’s Royal Society Open Science). Those editorial criteria have proved their worth.

But what are we going to call this style of peer-review?

It’s not a new problem. I discussed it with with David Crotty three years ago without reaching any very satisfactory conclusion. But three years have not really helped us much as we try to agree on a term for this increasingly important and prevalent model.

What are the options on the table?

PLOS ONE-style peer-review. It’s a cumbersome term, and it privileges PLOS ONE when that is now far from the only journal to use this approach to peer-review (and may not even have been first).

Peer-review Lite. A snide term coined by people who wanted PLOS ONE to fail. It’s not a good description, and it carries baggage.

Scientific peer-review. This one came up in the discussion with David Crotty, but it’s not really acceptable because it would leave us still needing a term for what the Open Library of Humanities does.

Objective peer-review. This is the term that was used at the Royal Society meeting at the start of this week — the idea being that you review objectively for the quality of the research, but don’t make a subjective judgement of its importance. Several people didn’t like this on the grounds that even the “objective” half is inevitably subjective.

Any others that I missed?

I don’t have a good solution to propose to this problem; but I think it’s getting more urgent that we do solve it. We have to have a simple, unambiguous, universally understood term to understand a model of peer-review that is becoming increasingly pervasive and may well end up as the dominant form of peer-review.

Plough in — comments are open!

Update, 6pm

Liz Wager asked a very similar question four years ago, over on the BMJ blog: what to call the journals that use this approach to peer-review. Terms that she mentions include:

  • “bias to publish” (from BioMed Central)
  • “non-selective” (her own coinage, which she doesn’t like)
  • “bumboat” (I can’t explain this one, you’ll have to read the article)
  • “author-driver” or “author-focused” publication (AFP for short)
  • “search-located” (which she coins, the dismisses as tautologous)
  • “unconventional” or “non-traditional” (discarded as disparaging)
  • “non-discriminatory”, “impartial” or “unprejudiced”
  • “general” (dismissed as a non-starter)
  • “broad-spectrum” (inapplicable to specialised journals)

And then in the comments various people proposed:

  • “below the fold” journals
  • “omnivorous” (I quite like that one)
  • “alternative”
  • “Voldermortian journals”, which I don’t understand at all.
  • “Unfiltered”, contrasted with “filtered”
  • “inclusive”, contrasted with “exclusive” (I quite like this, too)
  • “high volume low hassle”

But there’s no conclusion or preferred term.

The REF (Research Excellence Framework) is a time-consuming exercise that UK universities have to go through every few years to assess and demonstrate the value of their research to the government; the way funding is allocated between universities is largely dependent on the results of the REF. The exercise is widely resented, in part because the processes of preparing and reviewing the submissions are so time-consuming.

Dorothy Bishop has noted that results of REF assessments correlate strongly with departmental H-indexes (and suggested that we could save on the cost of future REFs by just using that H-index instead of the time-consuming peer-review process).

But it’s also been shown that H-index is strongly correlated with the simple number of publications. A seductive but naive conclusion would be: “we could just count publications for the next REF!”

But of course if we simply allocated research funding across universities on the basis of how many papers they produce, they would — they would have to — respond by simply racing to write more and more papers. Our already overloaded ability to assimilate new information would be further flooded. It’s a perverse incentive.

So this is a classic example of a very important general dictum: measure the thing you’re actually interested in, not a proxy. I don’t know if this dictum has been given a name yet, but it ought to be.

Measuring the thing we’re interested in is often difficult and time-consuming. But since only the thing we measure will ever be optimised for, we must measure the thing we want optimised — in this case, quality of research rather than quantity. That would still be true even if the correlation between REF assessments and departmental H-indexes was absolutely perfect. Because that correlation is an accident (in the philosophical sense); changing the circumstances will break that correlation.

No doubt all this reasoning is very familiar and painfully basic to people who have been working on the problem of metrics in assessment for years; to them, I apologise. For everyone else, I hope this comment provides some grains of insight.

[I originally wrote the initial form of this post as a comment on the Royal Society blog how should research and impact be assessed?, but I’m still waiting for it to be approved, so here it is.]

 

I’m at the Royal Society today and tomorrow as part of the Future of Scholarly Scientific Communication conference. Here’s the programme.

I’m making some notes for my own benefit, and I thought I might as well do them in the form of a blog-post, which I will continuously update, in case anyone else is interested.

I stupidly didn’t make notes on the first two speakers, but let’s pick up from the third:

Deborah Shorley, ex-librarian of Imperial College London

Started out by saying that she feels her opinion, as a librarian, is irrelevant, because librarians are becoming irrelevant. A pretty incendiary opening!

Important observations:

“Scientific communication in itself doesn’t matter; what matters is that good science be communicated well.”

And regarding the model of giving papers to publishers gratis, then paying them for the privilege of reading them:

“I can’t think of any other area where such a dopey business model pertains.”

(On which, see Scott Aaronson’s brilliant take on this in his review of The Access Principle — the article that first woke me up to the importance of open access.)

Shorey wants to bring publishing skills back in-house, to the universities and their libraries, and do it all themselves. As far as I can make out, she simply sees no need for specialist publishers. (Note: I do not necessarily endorse all these views.)

“If we don’t seize the opportunity, market forces will prevail. And market forces in this case are not pretty.”

Robert Parker, ex-head of publishing, Royal Society of Chemistry

Feels that society publishers allowed themselves to be overtaken by commercial publishers. Notes that when he started working for the RSC’s publishing arm, it was “positively dickensian”, using technology that would mostly have been familiar to Gutenberg. Failure to engage with authors and with technology allowed the commercial publishers to get ahead — something that is only now being redressed.

He’s talking an awful lot about the impact factors of their various journals.

My overall impression is that his perspective is much less radical than that of Deborah Shorley, wanting learned-society publishers to be better able to compete with the commercial publishers.

Gary Evoniuk, policy director at Glaxo Smith Klein

GSK submits 300-400 scientific studies for publication each year.

Although the rise of online-only journals means there is no good reason to not publish any finding, they still find that negative results are harder to get published.

“The paper journal, and the paper article, will soon be dead. This makes me a little bit sad.”

He goes further and wonders whether we need journal articles at all? When actual results are often available long before the article, is the context and interpretation that it provides valuable enough to be worth all the effort that’s expended on it? [My answer: yes — Ed.]

Discussion now follows. I probably won’t attempt to blog it (not least because I will want to participate). Better check out the twitter stream.

Nigel Shadbolt, Open Data Institute

Begin by reflecting on a meeting ten years ago, convened at Southampton by Stevan Harnad, on … the future of scholarly scientific communication.

Still optimistic about the Semantic Web, as I guess we more or less have to be. [At least, about many separate small-sw semantic webs — Ed.] We’re starting to see regular search-engines like Google taking advantage of available machine-readable data to return better results.

Archiving data is important, of course; but it’s also going to be increasingly important to archive algorithms. github is a useful prototype of this.

David Lambert, president/CEO, internet2

Given how the digital revolution has transformed so many fields (shopping, auctions, newspapers, movies) why has scholarly communication been so slow to follow? [Because the incumbents with a vested interesting in keeping things as they are have disproportionate influence due to their monopoly ownership of content and brands — Ed.]

Current publication models are not good at handling data. So we have to build a new model to handle data. In which case, why not build a new model to handle everything?

New “born-digital” researchers are influenced by the models of social networks: that is going to push them towards SN-like approaches of communicating more stuff, more often, in smaller unit. This is going to affect how scholarly communication is done.

Along with this goes an increasing level of comfort with collaboration. [I’m not sure I see that — Ed.]

Bonus section: tweets from Stephen Curry

He posted these during the previous talk. Very important:

Ritu Dhand, Nature

[A disappointing and unconvincing apologia for the continuing existence and importance of traditional publishers, and especially Nature. You would think that they, and they alone, guard the gates of academia from the barbarians. *sigh*. — Ed.]

Lunch

Georgina Mace, UCL

[A defence of classical peer-review. Largely an overview of how peer-review is supposed to work.]

“It’s not perfect, it has its challenges, but it’s not broken yet.”

Richard Smith, ex-editor of BMJ

[An attack on classical peer-review.]

“Peer review is faith-, not evidence-based; ineffective; a lottery; slow; expensive; wasteful; ineffective; easily abused; biased; doesn’t detect fraud; irrelevant.

Apart from that, it’s perfect.”

He doesn’t want to reform peer-review, he wants to get rid of it. Publish, let the world decide. That’s the real peer-review.

He cites studies supporting his assertions. Cochrane review concluded there is no evidence that peer-review is effective. The Ioannidis paper shows that most published findings are false.

Someone should be recording this talk. It’s solid gold.

Annual cost of peer-review is $1.9 billion.

[There is much, much more. I can’t get it down quickly enough.]

 Georgina Mace’s rebuttal

… amounts to contradicting Richard Smith’s evidence-supported statements, but she provides no evidence in support of her position.

Richard Smith’s counter-counter rebuttal

… cites a bunch more studies. This is solid. Solid.

For those who missed out, see Smith’s equally brutal paper Classical peer review: an empty gun. I find his conclusion (that we should just dump peer-review) emotionally hard to accept, but extremely compelling based on actual, you know, evidence.

Fascinating to hear the level of denial in the room. People really, really want to keep believing in peer-review, in spite of evidence. I understand that impulse, but I think it’s unbecoming in scientists.

The challenge for peer-review advocates is: produce evidence that it has value. No-one has responded to that.

Richard Sever, Cold Spring Harbour Press

Richard presents the BiorXive preprint server. Turns out it’s pronounced “bio-archive”, not “bye-orx-ive”.

Nothing in this talk will be new to regular SV-POW! readers, but he makes good, compelling points in favour of preprinting (which we of course agree with!)

Elizabeth Marincola, CEO, PLOS

PLOS is taking steps towards improving peer-review:

  • Use of article-level metrics
  • Moves towards open review
  • Move toward papers evolving over time, not being frozen at the point of publication
  • Better recognition of different kinds of contribution to papers
  • Intention to make submitted paper available to view before peer-review has been carried out, subject only to checks on ethical and technical standard: they aim to make papers available in “a matter of days”.

She notes that much of this is not original: elements of these approaches are in F1000 Research, BiorXiv, etc.

Jan Velterop, science publisher with everyone at some point.

“I’m basically with Richard Smith when it comes to abolishing peer review, but I have a feeling it won’t happen in the next few weeks.”

The situation of publishers:

“Academia throws money at you. What do you do? You pick it up.”

Velterop gets a BIG laugh for this:

“Does peer-review benefit science? I think it does; and it also benefits many other journals.”

He quotes a Scholarly Kitchen blog-post[citation needed] as saying that the cost of technical preparation at PubMed Central — translating from an MS-Word manuscript to valid JATS XML — at $47. So why do we pay $3000 APCs? Surely the peer-review phase doesn’t cost $2953?

Update: here is that Scholarly Kitchen article.

Velterop’s plan is to streamline the review-and-publish process as follows:

  • Author writes manuscript.
  • She solicits reviews from two experts, using her own knowledge of the field to determine who is suitably skilled.
  • They eventually sign off (perhaps after multiple rounds of revisions)
  • The author submits the manuscript, along with the endorsements.
  • The editor checks with the endorsers that they really have given endorsement.
  • The article is posted.

Bam, done!

And at that point in the proceedings, my battery was running dangerously low. I typed a tweet: “low battery may finally force me to shut up! #RSSC”, but literally between typing at and hitting the Tweet button, my laptop shut down. So that’s it for day 1. I’ll do a separate post for the second and final day.

There’s been some concern over Scientific Reports‘ new scheme whereby authors submitting manuscripts can pay $750 to have them peer-reviewed more quickly. Some members of the editorial board have quit over this development, feeling that it’s unfair to authors who can’t pay. Myself, I feel it at least shows admirable audacity — NPG has found a way to monetise its own lethargy, which is surely what capitalism is all about.

The real problem with this scheme is that $750 is an awful lot to gamble, as a sort of “pre-APC”, at a point when you don’t know whether your article is actually going to be published or not. If the peer-review returns an unfavourable verdict it’s just money down the drain.

So I welcome today’s announcement that, for only a slightly higher payment of a round $1000, it’s now possible to bypass peer-review completely, and move directly to publication. This seems like a much fairer deal for authors, and of course it streamlines the publication process yet further. Now authors can obtain the prestigious Nature Publishing Group imprint in a matter of a couple of days.

Onward and upward!

Arriving as an early Christmas present, and coming in just a week before the end of what would otherwise have been a barren 2014, my paper Quantifying the effect of intervertebral cartilage on neutral posture in the necks of sauropod dinosaurs is out! You can read it on PeerJ (or download the PDF).

Figure 4. Effect of adding cartilage to the neutral pose of the neck of Diplodocus carnegii CM 84. Images of vertebra from Hatcher (1901:plate III). At the bottom, the vertebrae are composed in a horizontal posture. Superimposed, the same vertebrae are shown inclined by the additional extension angles indicated in Table 2.

Figure 4: Effect of adding cartilage to the neutral pose of the neck of Diplodocus carnegii CM 84. Images of vertebra from Hatcher (1901:plate III). At the bottom, the vertebrae are composed in a horizontal posture. Superimposed, the same vertebrae are shown inclined by the additional extension angles indicated in Table 2.

Yes, that posture is ludicrous — but the best data we currently have says that something like this would have been neutral for Diplodocus once cartilage is taken into account. (Remember of course that animals do not hold their necks in neutral posture.)

The great news here is that PeerJ moved quickly. In fact here’s how the time breaks down since I submitted the manuscript (and made it available as a preprint) on 4 November:

28 days from submission to first decision
3 days to revise and resubmit
3 days to accept
15 days to publication

TOTAL 49 days

Which of course is how it ought to be! Great work here from handling editor Chris Noto and all three reviewers: Matt Bonnan, Heinrich Mallison and Eric Snively. They all elected not to be anonymous, and all gave really useful feedback — as you can see for yourself in the published peer-review history. When editors and reviewers do a job this good, they deserve credit, and it’s great that PeerJ’s (optional) open review lets the world see what they contributed. Note that you can cite, or link to, individual reviews. The reviews themselves are now first-class objects, as they should be.

At the time of writing, my paper is top of the PeerJ home-page — presumably just because it’s the most recent published paper, but it’s a nice feeling anyway!

Screenshot from 2014-12-23 10:39:34

 

A little further down the front-page there’s some great stuff about limb function in ratites — a whole slew of papers.

Well, I’m off to relax over Christmas. Have a good one, y’all!

Despite the flagrant trolling of its title, Nature‘s recent opinion-piece Open access is tiring out peer reviewers is mostly pretty good. But the implication that the rise of open-access journals has increased the aggregate burden of peer-review is flatly wrong, so I felt obliged to leave a comment explaining why. Here is that comment, promoted to a post of its own (with minor edits for clarity):


 

Much of what is said here is correct and important. Although it would be nice if Nature could make a bit more of an effort to avoid the obvious conflict-of-interest issues that lead it to title the piece so misleadingly as an attack on open access. I am glad that so many of the other commenters on this piece saw straight through that rather snide piece of propaganda.

Only one important error of interpretation here, I think. I quote:

The rise of the open-access (OA) movement compounds this effect [i.e. the increasing number of articles needing peer-review.] The business case for online OA journals, to which authors pay submission fees, works best at high volume. And for many of these journals, submitted work is published as long as it is methodologically sound. It does not have to demonstrate, for example, the novelty or societal relevance that some traditional journals demand.

The implication is that journals of this kind (PLOS ONE, PeerJ, the various Frontiers journals) increase the total peer-review burden. In fact, the exact opposite is the case. They greatly reduce the the total amount of peer reviewing.

It’s an open secret that nearly every paper eventually gets published somewhere. Under the old regime, the usual approach is to “work down the ladder”, submitting the same paper repeatedly to progressively less prestigious journals until it reached one that was prepared to publish work of the supplied level of sexiness. As a result, many papers go through four, five or more rounds of peer-review before finally finding a home. Instead, such papers when submitted to a review-for-soundness-only venue such as PLOS ONE require only a single round of review. (Assuming of course that they are indeed methodologically sound!)

The rise of review-for-soundness-only journals (“megajournals”) is an unequivocal improvement in the scientific publishing landscape, and should be welcomed by all parties: authors, who no longer have to submit to the monumental waste of time and effort that is the work-down-the-ladder system; readers, who get access to new research much more quickly; and editors and reviewers who no longer have to burn hours re-reviewing and re-re-reviewing perfectly good papers that have already been repeatedly rejected for a perceived lack of glamour.

Follow

Get every new post delivered to your Inbox.

Join 529 other followers