Sauropod Vertebra Picture of the Week

I’ll try to live-blog the first day of part 2 of the Royal Society’s Future of Scholarly Scientific Communication meeting, as I did for the first day of part 1. We’ll see how it goes.

Here’s the schedule for today and tomorrow.

Session 1: the reproducibility problem

Chair: Alex Halliday, vice-president of the Royal Society

Introduction to reproducibility. What it means, how to achieve it, what role funding organisations and publishers might play.

For an introduction/overview, see #FSSC – The role of openness and publishers in reproducible research.

Michele Dougherty, planetary scientist

It’s very humbling being at this meeting, when it’s so full of people who have done astonishing things. For example, Dougherty discovered an atmosphere around one of Saturn’s moons by an innovative use of magnetic field data. So many awesome people.

Her work is largely to do with very long-term project involving planetary probes, e.g. the Cassini-Huygens probe. It’s going to be interesting to know what can be said about reproducibility of experiments that take decades and cost billions.

“The best science output you can obtain is as a result of collaboration with lots of different teams.”

Application of reproducibility here is about making the data from the probes available to the scientific community — and the general public — so that the result of analysis can be reproduced. So not experimental replication.

Such data often has a proprietary period (essentially an embargo) before its public release, partly because it’s taken 20 years to obtain and the team that did this should get the first crack at it. But it all has to be made publicly available.

Dorothy Bishop, chair of Academy of Medical Sciences group on replicability

The Royal Society is very much not the first to be talking about replicability — these discussions have been going on for years.

About 50% of studies in Bishop’s field are capable of replication. Numbers are even worse in some fields. Replication of drug trials are particularly important, as false result kill people.

Journals cause awful problems with impact-chasing: e.g. high-impact journals will publish sexy-looking autism studies with tiny samples, which no reputable medical journal would publish.

Statistical illiteracy is very widespread. Authors can give the impression of being statistically aware but in a superficial way.

Too much HARKing going on (Hypothesising After Results Known — searching a dataset for anything that looks statistically significant in the shallow p < 0.05 sense.)

“It’s just assumed that people doing research, know what they are doing. Often that’s just not the case.”

… many more criticisms of how the journal system encourages bad research. They’re coming much faster than I can type them. This is a storming talk, I wish the record would be made available.

Employers are also to blame for prioritising expensive research proposals (= large grants) over good ones.

All of this causes non-replicable science.

Floor discussion

Lots of great stuff here that I just can’t capture, sorry. Best follow the tweet stream for the fast-moving stuff.

One highlight: Pat Brown thinks it’s not necessarily a problem if lots of statistically underpowered studies are performed, so long as they’re recognised as such. Dorothy Bishop politely but emphatically disagrees: they waste resources, and produce results that are not merely useless but actively wrong and harmful.

David Colhoun comments from the floor: while physical sciences consider “significant results” to be five sigmas (p < 0.000001), biomed is satisfied with slightly less than two sigmas (p < 0.05) which really should be interpreted only as “worth another look”.

Dorothy Bishop on publishing data, and authors’ reluctance to do so: “It should be accepted as a cultural norm that mistakes in data do happen, rather than shaming people who make data open.”

Coffee break

Nothing to report :-)

Session 2: what can be done to improve reproducibility?

Iain Hrynaszkiewicz, head of data, Nature

In an analysis of retractions of papers in PubMed Central, 2/3 were due to fraud and 20% due to error.

Access to methods and data is a prerequisite for replicability.

Pre-registration, sharing of data, reporting guidelines all help.

“Open access is important, but it’s only part of the solution. Openness is a means to an end.”

Hrynaszkiewicz says text-miners are a small minority of researchers. [That is true now, but I and others are confident this will change rapidly as the legal and technical barriers are removed: it has to, since automated reading is the only real solution to the problem of keeping up with an exponentially growing literature. — Ed.]

Matthew Cockerill, founder of BioMed Central

Fascinating slides. I hope he’ll post them. They’re moving too fast for me to get down the quotes on them, sorry!

Mostly relates to the rigorous approaches used to improve manufacturing and industrial processes: intolerance of variability leads to better understanding of confounding factors. Can we learn from this in research? [My take: not necessarily. By definition, manufacturing is about reliably applying existing knowledge, while research is about creating new knowledge. — Ed.]

Nicole Janz, University of Cambridge

Anecdotes of various authors’ failure to produce data for her when requested: people say they need to clean it up first, that they don’t have time, or that they don’t have it to hand.

Janz teaches the Cambridge Replication Workshop. She is working to change this culture by requiring good data practices from her students as part of their course.

Update: here is a transcript of her talk.

Floor discussion

I make the point that however well we teach students how good replicable practices, we’re doomed while they still believe they need to publish short, sexy, insubstantial papers in Science or Nature for their career. So once again, this come down to the problem of career incentives. (This seems to be broadly accepted across the room.)

Ritu Dhand of Nature makes the point that they accept long methods sections (presumably relegated to supplementary materials, but OK) — but very few authors take advantage of this option.

I respond: then don’t accept papers with short methods sections.

That got a laugh, but I didn’t want one. I was deadly serious. If Nature is serious about fixing the incentive problem it’s helped create, it must insist on rewarding only good scientific practice.

Lunch

Closing down for now … blog will resume in the afternoon.

Session 3: dealing with fraud and misconduct

Ottoline Leyser

Introduction to scientific fraud, and the vexed question of why anyone would do it.

This is the first time I’ve heard the phrase “single-word-title journals” — spoken with a subtle, but distinct sneer. Revisits the ubiquitious theme of perverse incentives: the perceived need to get into Science and Nature catalyses fraud.

James Parry, CEO, UK Research Integrity Office.

Notes that UKRIO is only a charity, and has no regulatory power. It’s there for people to go to for help and advice. About 100 people do so each year. It was initially established for biomed research, but now covers all areas of research.

Focussing on major cases of fraud (e.g. human cloning) is unproductive, because it obscures a much broader problem.

Researchers often don’t see integrity and misconduct as relevant to what they’re doing, e.g. postdocs may expect that their PI will handle “all that stuff”.

Although standards of research integrity seem obvious (tell the truth, don’t make up data, don’t harm participants, don’t embezzle research funds), in practice they can be awkward and demanding to follow.

Don’t be heavy handed and don’t micromanage.

Virginia Barbour, chair, Committee on Publication Ethics

What can publishers be doing to help address fraud?

COPE started in 1997 with the editors of three medical journals. Richard Smith (see part 1) was one of them. They now have ?3000 journal members. (Might have misheard the number: correct me if so.) They are a volunteer not-for-profit organisation.

“We are the canaries in the ethics mine.”

Barbour mentions COPE’s need to handle misconduct issues fairly and calmly despite “the online fury you often see”. So true. See Jon Ronson’s book.

Barbour once had a death threat related to her COPE activities. “At least, I think it was a death threat; it was very vague, and badly worded.”

“It is the core responsibility of journals and editors not to walk away from difficult issues.”

“Faking reviews seen by some as business transaction. Hiding data that don’t fit has become established practice.”

David Vaux, deputy director, Walter & Eliza Hall Institute

“I’ve been asked to be provocative, so I’ll try to be provocative without being offensive, and apologies in advance to Ritu [the Nature editor]”

He cites an example of a very over-precise reporting in Nature. Correction shows much larger standard deviations, and measuring tumours by volume rather than length. He wrote to Nature explaining this.

The published a correction, but the correction introduced its own problems, such as implying a 7cc tumour in a 20g mouse, which greatly exceeds what is ethically permissible in animal experimentation. (Guidelines say that mouse with a tumour reaching .5g must immediately be euthanised.) The photos of the mice are truly grotesque.

He wrote to Nature complaining about the correction. It’s been open since October 2012, and they have still not decided how to proceed.

Vaux says that Journals should publish correction more readily and get it done much more quickly; and they should separate the correction aspect from the punishment aspect.

Floor Discussion

Several people, including Dorothy Bishop, make the case once more for post-publication peer-review — which was one of the big winners in part 1 of the meeting.

Bishop points out that it’s better to raise issues with papers out in the open, rather than directly with the editor: we see this in the much quicker responses, and if the fact that the problem can’t be hushed up. [Example, PLOS sat on the sexist review until the victim started tweeting about it. — Ed.]

David Vaux points out that, especially in the USA, the PI doesn’t have to fake anything – they have 50 postdocs to compete to prove that their boss’s pet hypothesis is right, and the first to do so gets tenure. In a race like that, there will be misconduct.

Ginny Barbour says that if editors identify a problem with a paper, it’s not good enough to reject and then walk away — they have to engage with the issue and pursue it.

James Parry points out that you often hear investigations resulting in “We have spoken to the author in question and are satisfied with their response.” Not good enough.

Like so very, very much, this discussion is converging on the problem of perverse incentives. David Colquhoun reminds us that an Imperial researcher actually killed himself because of a perverse incentive structure. Has to change. He was told he would lose his job if he didn’t get in £200k in grants. How could such pressure not result in misconduct?

People pursue flaws in science as a hobby, says Ben Goldacre. Should learned societies fund figures who are ‘public prosecutors’? Virginia Barbour thinks no: it will do nothing for the much more pervasive problem of “sloppiness”.

David Vaux (from the panel) thinks that the biggest thing we could do to address the problem of fraud is introducing double-blind peer-review, because much publication on bad science comes from papers being accepted on the basis of who the author is rather than whether the work is any good.

Coffee break

… nothing to report …

Session 4: the importance of scientific integrity

Mark Wolport, government chief scientific advisor

There is a spectrum of bad science:

• wickedness
• incompetence
• unluckiness

Dealing with wickedness (deliberate fraud) requires punishment in part as a deterrent. Education also important: on values, but also on competence.

“The publication system includes loads of perverse incentives. There’s a publication bias against studies that contradict, or indeed confirm earlier work. And scientists obsession with journal brand inflames these problems.”

That last quote (really a paraphrase) is not news to those who have been following these meetings, but it’s good to hear it stated so clearly be someone so influential.

There is a problem of authors becoming biased advocates of their own work when doing public engagement.

“We must move towards preprints, v1.0 of a paper, and the accretion of further evidence and validation in v2.0, v3.0.” [And what about v1.1, incremental improvements? — Ed.]

(That was quite a short talk, and leads directly into discussion.)

Floor discussion

Pay Brown reminds us that all published results are subject to further analysis and correction, whether peer-reviewed or not. See Peer review does not mean we can trust a published paper.

David Colquhoun thinks that the solution to many our ills is to abolish all metrics. I like his idealism, but even I think that is hopelessly unrealistic.

Ben Goldacre argues that press-releases should be published alongside papers, to be subjected to similar post-publication critique. But Walport is concerned that we avoid erecting yet another big bureaucratic framework around press communications.

Mark Thorley (RCUK) points out that improving reliability, statistical power, etc., means more work. If we invest in this, it will mean that the UK publishes fewer but better papers. Will that be worth the trade-off?

Mark Walport thinks we need to get away from the model where researchers are always writing new papers (often with the same introductions), towards writing one paper and incrementally revising it. [I am sceptical. That way you get one set of citations, the old way you get n sets, and *promotion*. Once again, it’s all about incentives. — Ed.]

Ben Goldacre says we need funding streams for experimental projects trying out all the ideas being suggested today. [I strongly agree! But then I am biased. — Ed.]

And that’s all, folks …

Dragons’ Den next, but I’m one of the participants so I won’t be blogging or tweeting. Hopefully others will take up the slack!