It’s a bit shocking to find that SV-POW! is going on for twelve years old. (Our very first post was on 1st October 2007, so we’re about fifty days short of that anniversary.)

It’s cost us almost nothing to run the blog in that time — in financial terms, at least. We pay $18 a year for the domain svpow.com (as opposed to svpow.wordpress.com), and that’s it. We’ve been happy to pay out of our own pockets, because not taking any money from anyone means we don’t owe anyone anything and we can write what we like, on whatever subject we like, as often or as infrequently as we like. It’s a great way to do things.

But there are a couple more things we’d like to do.

One is to get rid of the adverts that infest the site. I wasn’t aware of them until recently, when I looked at the site on a new phone. They’re pretty intrusive, and I remember reading somewhere that they’ve got a lot worse recently. We can do that for $7 a month, which comes to $84 per year. That’s enough that I can’t just shrug like I do with the $18 and say “Ah, that’s the price of a round of drinks, just pay it.”

But more excitingly, Matt and I would like to get together more often for scientific trips like the 2016 Sauropocalypse and this year’s visit to the Carnegie museum. These trips are amazingly productive and generate a ton of observations, photos and videos, which we’re then able to turn into science, which in turn becomes papers. But because Matt and I are inconveniently located 5,000 miles and eight timezones apart, it’s logistically difficult and expensive for us to get together. That’s why we had a three-year gap between the Utah and Carnegie trips, and we’d like — if we can — to do something like this every year.

Here I am, in the collections of the Carnegie Museum, Pittsburgh, earlier this year. I’m documenting the re-articulated centrum, neural arch and cervical ribs of C6 of the sub-adult apatosaurine CM 555, probably Brontosaurus parvus.

Why are these trips important to do? Because we’ve found that when both of us look at bones together, we see much more than twice as much as either of us would see alone. There seems to be a subtle alchemy at play here, like the way Lennon and McCartney achieved so much more when they were working together than they did after the Beatles split(*). Somehow, there is a sparking that happens, as each of us amplifies and re-interprets the other’s observations. (Probably the same thing happens for any two palaeontologists looking at bones together, but Matt and I have got into the habit of working together.)

To do more of this, we need more money: Matt’s travel funds are limited; mine are literally non-existent, and I have to fund all my own work out of money that I earn by the sweat of my brow (and burn up precious vacation days from my real job).

That’s why we’ve finally created a Patreon page where anyone who wants to can sign up to support the work we do here. We have no idea whether this will work out, but whatever money might accumulate, we will spend first on the basic running costs of the site ($18 + $84 = $102 per year), and then on research trips. The big-ticket items are travel (including a transatlantic flight for one of us) and accomodation. After that, it’s just food and drink, which we pay for ourselves since we’d be eating and drinking anyway.

What do you get in return for your money? Absolutely nothing. This is really important. It goes back to what I said earlier about how valuable it’s been to us that we are free to write what we want, when we want. We’re not going to sacrifice that freedom for money, and in fact the value of that freedom is exactly why we’ve never solicited any kind of donation before. We want to be 100% clear that basic-level subscribers get nothing for their money. No early access, no extra content, no physical rewards. Just the knowledge that you’re supporting a blog that you like and helping to make sauropod science happen. It will only take nine of you to subscribe at the base $1-per-month level to pay our WordPress fees.

But for anyone who chooses to support the SV-POW! research trips at the level of $5 per month or higher, there is one small reward: we’ll then consider you as a research funder, and will formally mention you as such in the acknowledgements of the first paper that comes out of the first research trip we do after you sign up. That’s a place — admittedly a small one — in the permanent scientific record. Plus of course you get our gratitude!

For those who don’t sign up, absolutely nothing will change. We’ll continue our usual erratic schedule of posting whatever happens to occur to us, when it occurs to us. You’ll still get immediate access to everything we write for the blog, just like those who do sign up.

So: if you would like to help finance the scientific mission of SV-POW!, get yourself over to https://www.patreon.com/svpow

 

 


Note

(*) “But Mike”, you say, “After the first few years, the Lennon-McCartney partnership was in name only, and they hardly contributed to each other’s songs”. And you’re right: while We Can Work It Out was a true 50-50 collaboration, lots of the later songs were essentially solo compositions. But here’s the thing: even then, the mere presence of the other partner pushed them to up their game. John Lennon may have made no concrete contribution to Hey Jude (beyond telling Paul that he should keep his placeholder lyric “The movement you need is on your shoulder”), but McCartney only landed that song as perfectly as he did because Lennon was around. If you want proof, look at his post-Beatles songwriting output. There are plenty of good songs, but almost no great ones: Maybe I’m Amazed would be the strongest contender, and it turns out (I just this moment looked it up on Wikipedia) that he wrote that before the Beatles broke up. In a similar way, I wrote all but a couple of paragraphs of the actual prose in our in-progress vertebral orientation manuscript, but it all emerged from discussion between us, and could never have happened without Matt. He’s my John Lennon. But he lives on a different continent. Dammit.

We as a community often ask ourselves how much it should cost to publish an open-access paper. (We know how much it does cost, roughly: typically $3000 with a legacy publisher, or an average of $900 with a born-open publisher, or nothing at all for many journals.)

We know that peer-review is essentially free to publishers, being donated free by scholars. We know that most handling editors also work for free or for peanuts. We know that hosting things on the Web is cheap (“publishing [in this sense] is just a button“).

Publishers have costs associated with rejecting manuscripts — checking that they’re by real people at real institutions, scanning for obvious pseudo-scholarship, etc. But let’s ignore those costs for now, as being primarily for the benefit of the publishers rather than the author. (When I pay a publisher an APC, they’re not serving me directly by running plagiarism checks.)

The tendency of many discussions I’ve been involved with has been that the main technical contribution of publishers is the process that is still, for historical reasons, known as “typesetting” — that is, the transformation of the manuscript from from an opaque form like an MS-Word file (or indeed a stack of hand-written sheets) into a semantically rich representation such as JATS XML. From there, actual typesetting into HTML or a pretty PDF can be largely automated.

So: what does it cost to typeset a manuscript?

First data point: I have heard that Kaveh Bazargan’s River Valley Technologies (the typesetter that PeerJ and many more mainstream publishers use) charges between £3.50 and £9 per page, including XML, graphics, PDF generation and proof correction.

Second data point: in a Scholarly Kitchen post that Kent Anderson intended as a criticism of PubMed Central but which in fact makes a great case for what good value it provides, he quotes an email from Kent A. Smith, a former Deputy Director of the NLM:

Under the % basis I am using here $47 per article. John [Mullican, a program analyst at NCBI] and I looked at this yesterday and based the number on a sampling of a few months billings. It consists on the average of about $34-35 per tagged article plus $10-11 for Q/A plus administrative fees of $2-3, where applicable.

Using the quoted figure of $47 per PMC article and the £6.25 midpoint of River Valley’s range of per-page prices (= $9.68 per page), that would be consistent with typical PMC articles being a bit under five pages long. The true figure is probably somewhat higher — maybe twice as long or more — but this seems to be at least in the same ballpark.

Third data point: Charles H. E. Ault, in a comment on that Scholarly Kitchen post, wrote:

As a production director at a small-to-middling university press that publishes no journals, I’m a bit reluctant to jump into this fray. But I must say that I am astonished at how much PMC is paying for XML tagging. Most vendors looking for the small amount of business my press can offer (say, maybe 10,000 pages a year at most) charge considerably less than $0.50 per page for XML tagging. Assuming a journal article is about 30 pages long, it should cost no more than $15 for XML tagging. Add another few bucks for quality assurance, and you might cross the $20 threshold. Does PMC have to pay a federally mandated minimum rate, like bridge construction projects? Where can I submit a bid?

I find the idea of 50-cent-per-page typesetting hard to swallow — it’s more than an order of magnitude cheaper than the River Valley/PMC level, and I’d like to know more about Ault’s operation. Is what they’re doing really comparable with what the others are doing?

Are there other estimates out there?

 

I hate to keep flogging a dead horse, but since this issue won’t go away I guess I can’t, either.

1. Two years ago, I wrote about how you have to pay to download Elsevier’s “open access” articles. I showed how their open-access articles claimed “all rights reserved”, and how when you use the site’s facilities to ask about giving one electronic copy to a student, the price is £10.88. As I summarised at the time: “Free” means “we take the author’s copyright, all rights are reserved, but you can buy downloads at a 45% discount from what they would otherwise cost.” No-one from Elsevier commented.

2. Eight months ago, Peter Murray-Rust explained that Elsevier charges to read #openaccess articles. He showed how all three of the randomly selected open-access articles he looked at had download fees of $31.50. No-one from Elsevier commented (although see below).

3. A couple of days ago, Peter revisited this issue, and found that Elsevier are still charging THOUSANDS of pounds for CC-BY articles. IMMORAL, UNETHICAL , maybe even ILLEGAL.This time he picked another Elsevier OA article at random, and was quoted £8000 for permission to print 100 copies. The one he looked at says “Open Access” in gold at the top and “All rights reserved” at the bottom. Its “Get rights and content” link takes me to RightsLink, where I was quoted £1.66 to supply a single electronic copy to a student on a course at the University of Bristol:

Screenshot from 2014-03-11 09:40:35

(Why was I quoted a wildly different price from Peter? I don’t know. Could be to do with the different university, or because he proposed printing copies instead of using an electronic one.)

On Peter’s last article, an Elsevier representative commented:

Alicia Wise says:
March 10, 2014 at 4:20 pm
Hi Peter,

As noted in the comment thread to your blog back in August we are improving the clarity of our OA license labelling (eg on ScienceDirect) and metadata feeds (eg to Rightslink). This is work in progress and should be completed by summer. I am working with the internal team to get a more clear understanding of the detailed plan and key milestones, and will tweet about these in due course.

With kind wishes,

Alicia

Dr Alicia Wise
Director of Access and Policy
Elsevier
@wisealic

(Oddly, I don’t see the referenced comment in the August blog-entry, but perhaps it was on a different article.)

Now here is my problem with this.

First of all, either this is deliberate fraud on Elsevier’s part — charging for the use of something that is free to use — or it’s a bug. Following Hanlon’s razor, I prefer the latter explanation. But assuming it’s a bug, why has it taken two years to address? And why is it still not fixed?

Elsevier, remember, are a company with an annual revenue exceeding £2bn. That’s £2,000,000,000. (Rather pathetically, their site’s link to the most recent annual report is broken, but that’s a different bug for a different day.) Is it unreasonable to expect that two years should be long enough for them to fix a trivial bug?

All that’s necessary is to change the “All rights reserved” message and the “Get rights and content” link to say “This is an open-access article, and is free to re-use”. We know that the necessary metadata is there because of the “Open Access” caption at the top of the article. So speaking from my perspective as a professional software developer of more than thirty years’ standing, this seems like a ten-line fix that should take maybe a man-hour; at most a man-day. A man-day of programmer time would cost Elsevier maybe £500 — that is, 0.000025% of the revenue they’ve taken since this bug was reported two years ago. Is it really too much to ask?

(One can hardly help comparing this performance with that of PeerJ, who have maybe a ten-thousandth of Elsevier’s income and resources. When I reported three bugs to them in a course of a couple of days, they fixed them all with an average report-to-fix time of less than 21 hours.)

Now here’s where it turns sinister.

The PeerJ bugs I mentioned above cost them — not money, directly, but a certain amount of reputation. By fixing them quickly, they fixed that reputation damage (and indeed gained reputation by responding so quickly). By contrast, the Elsevier bug we’re discussing here doesn’t cost them anything. It makes them money, by misleading people into paying for permissions that they already have. In short, not fixing this bug is making money for Elsevier. It’s hard not to wonder: would it have remained unfixed for two years if it was costing them money?

But instead of a rush to fix the bug, we have this kind of thing:

I find that very hard to accept. However complex your publishing platform is, however many different modules interoperate, however much legacy code there is — it’s not that hard to take the conditional that emits “Open Access” in gold at the top of the article, and make the same test in the other relevant places.

As John Mark Ockerbloom observes:

Come on, Elsevier. You’re better than this. Step up. Get this done.

Update (21st March 2014)

Ten days layer, Elsevier have finally responded. To give credit where it’s due, it’s actually pretty good: it notes how many customers made payments they needn’t have made (about 50), how much they paid in total (about $4000) and says that they are actively refunding these payments.

It would be have been nice, mind you, had this statement contained an actual apology: the words “sorry”, “regret” and “apologise” are all notably absent.

And I remain baffled that the answer to “So when will this all be reliable?” is “by the summer of 2014”. As noted above, the pages in question already have the information that the articles are open access, as noted in the gold “Open Access” text at top right of the pages. Why it’s going to take several more months to use that information elsewhere in the same pages is a mystery to me.

Update 2 (24th March 2014)

As noted by Alicia in a comment below, Elsevier employee Chris Shillum has posted a long comment on Elsevier’s response, explaining in more detail what the technical issues are. Unfortunately there seems to be no way to link directly to the comment, but it’s the fifth one.

 

From the files of J. K. Rowling.

Publisher #1

Dear Ms. Rowling,

Thank you for submitting your manuscript Harry Potter and the Half-Blood Prince. We will be happy to consider it for publication. However we have some concerns about the excessive length of this manuscript. We usually handle works of 5-20 pages, sometimes as much as 30 pages. Your 1337-page manuscript exceeds these limits, and requires some trimming.

We suggest that this rather wide-ranging work could usefully be split into a number of smaller, more tightly focussed, papers. In particular, we feel that the “magic” theme is not appropriate for our venue, and should be excised from the current submission.

Assuming you are happy to make these changes, we will be pleased to work with you on this project.

Correspondence ends.

Publisher #2

Esteemed Joenne Kay Rowling,

We are delightful to recieve your manuscript Harry Potter and the Half-Blood Prince and we look forword to publish it in our highly prestigious International Journal of Story Peer Reviewed which in 2013 is awarded an impact factor of 0.024.

Before we can progression this mutually benefit work, we require you to send a cheque for $5,000 US Dollars to the above address.

Correspondence ends.

Publisher #3

Dear J.R.R. Rowling,

We are in receipt of your manuscript Harry Potter and the Half-Blood Prince. Unfortunately, after a discussion with the editorial board, we concluded that it is insufficiently novel to warrant publication in our journal, which is one of the leading venues in its field. Although your work is well executed, it does not represent a significant advance in scholarship.

That is not to say that minor studies such as yours are of no value, however! Have you considered one of the smaller society journals?

Correspondence ends.

Publisher #4

Dear Dr. Rowling

Your submission Harry Potter and the Half-Blood Prince has passed initial editorial checks and will now be sent to two peer-reviewers. We will contact you when we have their reports and are able to make a decision.

Dear Dr. Rowling

Re: Harry Potter and the Half-Blood Prince.

We agree that eighteen months is too long for a manuscript to spend in review. On making inquiries, we find that we are unfortunately no longer able to contact the editor who was handling your submission.

We have appointed a new handling editor, who will send your submission to two new reviewers. We will contact you as soon as the new editor has made a decision.

Dear Dr. Rowling

Re: Harry Potter and the Half-Blood Prince.

Your complaint is quite justified. We will chase the reviewers.

Dear Dr. Rowling

I am pleased to say that the reviewers have returned their reports on your submission Harry Potter and the Half-Blood Prince and we are able to make an editiorial decision, which is ACCEPT WITH MAJOR REVISION.

Reviewer 1 felt that the core point of your contribution could be made much more succinctly, and recommended that you remove the characters of Ron, Hermione, Draco, Hagrid and Snape. I concur with his assessment that the final version will be tighter and stronger for these cuts, and am confident that you can make them in a way that does not compromise the plot.

Reviewer 2 was positive over all, but did not like being surprised by the ending, and felt that it should have been outlined in the abstract. She also felt that citation of earlier works including Lewis (1950, 1951, 1952, 1953, 1954, 1955, 1956) and Pullman (1995, 1997, 2000) would be appropriate, and noted an over-use of constructions such as “… said Hermione, warningly”.

Dear Dr. Rowling

Thank you for your revised manuscript of Harry Potter and the Half-Blood Prince, which it is our pleasure to accept. We now ask you to sign the attached copyright transfer form, so we can proceed with publication.

Dear Dr. Rowling

I am sorry that you are unhappy about this, but transfer of copyright is our standard procedure, and we must insist on it as a prerequisite for publication. None of our other authors have complained.

Dear Dr. Rowling

Thank you for the signed copyright transfer form.

In answer to your query, no, we do not pay royalties.

Dear Dr. Rowling

Sadly, no, we are unable to make an exception in the matter of royalties.

Dear Dr. Rowling

Your book has now been formatted. We attach a proof PDF. Please read this very carefully as this is the last chance to spot errors.

You will readily appreciate that publishing is an expensive business. In order to remain competitive we have had to reduce costs, and as a result we are no longer able to offer proof-reading or copy-editing. Therefore you are responsible for ensuring the copy is clean.

At this stage, changes should be kept as small as possible, otherwise a charge may be incurred for re-typesetting.

Dear Dr. Rowling

Many thanks for returning the corrected proofs of Harry Potter and the Half-Blood Prince. We will proceed with publication.

Now that the final length of your contribution is known, we are able to assess page charges. At 607 pages, this work exceeds our standard twenty free pages by 587. At $140 US per page, this comes to $82,180. We would be grateful if you would forward us a cheque for this amount at your convenience.

Dear Dr. Rowling

Thank you for you prompt payment of the page charges. We agree that these are regrettable, but sadly they are part of the reality of the publishing business.

We are delighted to inform you that Harry Potter and the Half-Blood Prince is now published online, and has been assigned the DOI 10.123.45678.

We thank you for working on this fine contribution with us, and hope you will consider us for your future publications.

Dear Dr. Rowling

You are correct, your book is not freely downloadable. As we explained earlier in this correspondence, publishing is an expensive business. We recover our substantial costs by means of subscriptions and paid downloads.

In our experience, those with the most need to read your book will probably have institutional access. As for those who do not: if your readers are as keen as you say, they will no doubt find the customary download fee of $37.95 more than reasonable. Alternatively, readers can rent online access at the convenient price of $9.95 per 24 hours.

Dear Dr. Rowling

I am sorry that you feel the need to take that tone. I must reiterate, as already stated, that the revenues from download charges are not sufficient for us to be able to pay royalties. The $37.95 goes to cover our own costs.

If you wish for your book to be available as “open access”, then you may take advantage of our Freedom Through Slavery option. This will attract a further charge of $3,000, which can be paid by cheque as previously.

Dr. Rowling

Your attitude is really quite difficult to understand. All of this was quite clearly set out on our web-site, and should have been understood by you before you made your submission.

As stated in the copyright transfer form that you signed, you do not retain the right to post freely downloadable copies of your work, since you are no longer the copyright holder.

Dr. Rowling

We must ask you not to contact your handling editor directly. He was quite shaken by your latest outburst. If you feel you must write to us again, we must ask you to moderate your language, which is quite unsuitable for a lady. Meanwhile, we remind you that our publishing agreement follows industry best practice. It’s too late to complain about it now.

Correspondence ends.

IP Lawyer #1

Dear Pyramid Web-Hosting,

Copyright claim

We write on behalf of our client, Ancient Monolith Scholarly Publishing, who we assert are the copyright holders of Harry Potter and the Half-Blood Prince. It has come to our attention that a copy of this copyrighted work has been posted on a site hosted by you at the URL below.

This letter is official notification under the provisions of Section 512(c) of the Digital Millennium Copyright Act (“DMCA”) to effect removal of the above-reported infringement. We request that you immediately issue a cancellation message as specified in RFC 1036 for the specified posting and prevent the infringer, Ms. J. K. Rowling, from posting the infringing material to your servers in the future. Please be advised that law requires you, as a service provider, to “expeditiously remove or disable access to” the infringing material upon receiving this notice. Noncompliance may result in a loss of immunity for liability under the DMCA.

Please send us at the address above a prompt response indicating the actions you have taken to resolve this matter.

Correspondence ends.

Historical Note

Examination of Ms. Rowling’s personal effects established that she had written most of a seventh book, Harry Potter and the Deathly Hallows. However, Rowling never sought to publish this final book in the series.

I was astonished yesterday to read Understanding and addressing research misconduct, written by Linda Lavelle, Elsevier’s General Counsel, and apparently a specialist in publication ethics:

While uncredited text constitutes copyright infringement (plagiarism) in most cases, it is not copyright infringement to use the ideas of another. The amount of text that constitutes plagiarism versus ‘fair use’ is also uncertain — under the copyright law, this is a multi-prong test.

So here (right in the first paragraph of Lavelle’s article) we see copyright infringement equated with plagiarism. And then, for good measure, the confusion is hammered home by the depiction of fair use (a defence against accusations of copyright violation) depicted as a defence against accusations of plagiarism.

This is flatly wrong. Plagiarism and copyright violation are not the same thing. Not even close.

First, plagiarism is a violation of academic norms but not illegal; copyright violation is illegal, but in truth pretty ubiquitous in academia. (Where did you get that PDF?)

Second, plagiarism is an offence against the author, while copyright violation is an offence against the copyright holder. In traditional academic publishing, they are usually not the same person, due to the ubiquity of copyright transfer agreements (CTAs).

Third, plagiarism applies when ideas are copied, whereas copyright violation occurs only when a specific fixed expression (e.g. sequence of words) is copied.

Fourth, avoiding plagiarism is about properly apportioning intellectual credit, whereas copyright is about maintaining revenue streams.

Let’s consider four cases (with good outcomes in green and bad ones in red):

  1. I copy big chunks of Jeff Wilson’s (2002) sauropod phylogeny paper (which is copyright the Linnean Society of London) and paste it into my own new paper without attribution. This is both plagiarism against Wilson and copyright violation against the Linnean Society.
  2. I copy big chunks of Wilson’s paper and paste it into mine, attributing it to him. This is not plagiarism, but copyright violation against the Linnean Society.
  3. I copy big chunks of Rigg’s (1904) Brachiosaurus monograph (which is out of copyright and in the public domain) into my own new paper without attribution. This is plagiarism against Riggs, but not copyright violation.
  4. I copy big chunks of Rigg’s paper and paste it into mine with attribution. This is neither plagiarism nor copyright violation.

Plagiarism is about the failure to properly attribute the authorship of copied material (whether copies of ideas or of text or images). Copyright violation is about failure to pay for the use of the material.

Which of the two issues you care more about will depend on whether you’re in a situation where intellectual credit or money is more important — in other words, whether you’re an author or a copyright holder. For this reason, researchers tend to care deeply when someone plagiarises their work but to be perfectly happy for people to violate copyright by distributing copies of their papers. Whereas publishers, who have no authorship contribution to defend, care deeply about copyright violation.

One of the great things about the Creative Commons Attribution Licence (CC By) is that it effectively makes plagiarism illegal. It requires that attribution be maintained as a condition of the licence; so if attribution is absent, the licence does not pertain; which means the plagiariser’s use of the work is not covered by it. And that means it’s copyright violation. It’s a neat bit of legal ju-jitsu.

References

  • Riggs, Elmer S. 1904. Structure and relationships of opisthocoelian dinosaurs. Part II, the Brachiosauridae. Field Columbian Museum, Geological Series 2:229-247, plus plates LXXI-LXXV.
  • Wilson, Jeffrey A. 2002. Sauropod dinosaur phylogeny: critique and cladistic analysis. Zoological Journal of the Linnean Society 136:217-276.

In the last few weeks, it’s been my pleasure and privilege to give invited talks on open access to both UCL and the University of Ulster. (Both of them went well, thanks for asking.)

Now they come to process expenses, and both universities have asked for scans of my passport. I explained to UCL that I was only expecting expenses, not a fee, and they backed down; but Ulster are very kindly giving me a fee, and contact there insists that “our Finance Office will insist on receiving this [passport scan] before they will process payment”.

That seems bizarre to me.

Has anyone else run into this?

Has anyone else been reluctant to comply? To me it seems like a strange intrusion, and a completely unnecessary violation of privacy. Either they want to pay me or they don’t — either is fine (I didn’t accept the Ulster invitation for the money). But I don’t see what my passport has to do with anything.

Am I being unreasonable? Or are they?

What should I do?

 

It’s well worth reading this story about Thomas Herndon, a graduate student who as part of his training set out to replicate a well-known study in his field.

The work he chose, Growth in a Time of Debt by Reinhart and Rogoff, claims to show that “median growth rates for countries with public debt over roughly 90 percent of GDP are about one percent lower than otherwise; average (mean) growth rates are several percent lower.” It has been influential in guiding the economic policy of several countries, reaffirming an austerity-based approach.

So here is Lesson zero, for policy makers: correllation is not causation.

To skip ahead to the punchline, it turned out that Reinhart and Rogoff made a trivial but important mechanical mistake in their working: they meant to average values from 19 rows of their spreadsheet, but got the formula wrong and missed out the last five. Those five included three countries which had experienced high growth while deep in debt, and which if included would have undermined the conclusions.

Therefore, Lesson one, for researchers: check your calculations. (Note to myself and Matt: when we revise the recently submitted Taylor and Wedel paper, we should be careful to check the SUM() and AVG() ranges in our own spreadsheet!)

Herndon was able to discover this mistake only because he repeatedly hassled the authors of the original study for the underlying data. He was ignored several times, but eventually one of the authors did send the spreadsheet. Which is just as well. But of course he should never have had to go chasing the authors for the spreadsheet because it should have been published alongside the paper.

Lesson two, for researchers: submit your data alongside the paper that uses it. (Note to myself and Matt: when we submit the revisions of that paper, submit the spreadsheets as supplementary files.)

Meanwhile, governments around the world were allowing policy to be influenced by the original paper without checking it — policies that affect the disposition of billions of pounds. Yet the paper only got its post-publication review because of an post-grad student’s exercise. That’s insane. It should be standard practice to have someone spend a day or two analysing a paper in detail before letting it have such a profound effect.

And so Lesson three, for policy makers: replicate studies before trusting them.

Ironically, this may be a case where the peer-review system inadvertently did actual harm. It seems that policy makers may have shared the widespread superstition that peer-reviewed publications are “authoritative”, or “quality stamped”, or “trustworthy”. That would certainly explain their allowing it to affect multi-billion-pound policies without further validation. [UPDATE: the paper wasn’t peer-reviewed after all! See the comment below.]

Of course, anyone who’s actually been through peer-review a few times knows how hit-and-miss the process is. Only someone who’s never experienced it directly could retain blind faith in it. (In this respect, it’s a lot like cladistics.)

If a paper has successfully made it through peer-review, we should afford it a bit more respect than one that hasn’t. But that should never translate to blind trust.

In fact, let’s promote that to Lesson four: don’t blindly trust studies just because they’re peer-reviewed.