Check your calculations. Submit your data. Replicate.

April 20, 2013


It’s well worth reading this story about Thomas Herndon, a graduate student who as part of his training set out to replicate a well-known study in his field.

The work he chose, Growth in a Time of Debt by Reinhart and Rogoff, claims to show that “median growth rates for countries with public debt over roughly 90 percent of GDP are about one percent lower than otherwise; average (mean) growth rates are several percent lower.” It has been influential in guiding the economic policy of several countries, reaffirming an austerity-based approach.

So here is Lesson zero, for policy makers: correllation is not causation.

To skip ahead to the punchline, it turned out that Reinhart and Rogoff made a trivial but important mechanical mistake in their working: they meant to average values from 19 rows of their spreadsheet, but got the formula wrong and missed out the last five. Those five included three countries which had experienced high growth while deep in debt, and which if included would have undermined the conclusions.

Therefore, Lesson one, for researchers: check your calculations. (Note to myself and Matt: when we revise the recently submitted Taylor and Wedel paper, we should be careful to check the SUM() and AVG() ranges in our own spreadsheet!)

Herndon was able to discover this mistake only because he repeatedly hassled the authors of the original study for the underlying data. He was ignored several times, but eventually one of the authors did send the spreadsheet. Which is just as well. But of course he should never have had to go chasing the authors for the spreadsheet because it should have been published alongside the paper.

Lesson two, for researchers: submit your data alongside the paper that uses it. (Note to myself and Matt: when we submit the revisions of that paper, submit the spreadsheets as supplementary files.)

Meanwhile, governments around the world were allowing policy to be influenced by the original paper without checking it — policies that affect the disposition of billions of pounds. Yet the paper only got its post-publication review because of an post-grad student’s exercise. That’s insane. It should be standard practice to have someone spend a day or two analysing a paper in detail before letting it have such a profound effect.

And so Lesson three, for policy makers: replicate studies before trusting them.

Ironically, this may be a case where the peer-review system inadvertently did actual harm. It seems that policy makers may have shared the widespread superstition that peer-reviewed publications are “authoritative”, or “quality stamped”, or “trustworthy”. That would certainly explain their allowing it to affect multi-billion-pound policies without further validation. [UPDATE: the paper wasn’t peer-reviewed after all! See the comment below.]

Of course, anyone who’s actually been through peer-review a few times knows how hit-and-miss the process is. Only someone who’s never experienced it directly could retain blind faith in it. (In this respect, it’s a lot like cladistics.)

If a paper has successfully made it through peer-review, we should afford it a bit more respect than one that hasn’t. But that should never translate to blind trust.

In fact, let’s promote that to Lesson four: don’t blindly trust studies just because they’re peer-reviewed.

9 Responses to “Check your calculations. Submit your data. Replicate.”

  1. Mickey Mortimer Says:

    From that link- “Reinhart and Rogoff are standing by their conclusions, despite the error.”

    Never let data get in the way of your hypothesis… :|

  2. Mike Taylor Says:

    To be fair to R&R, I don’t know how many other lines of evidence that paper also had pointing to the same conclusion.

  3. Matt Says:

    And make sure to watch out for stray commas so that you don’t add zeros into
    your formulas inadvertently, especially after editing. AVERAGE(A1,A2) does not equal AVERAGE(A1,A2,).

  4. Nathan Myers Says:

    researchers: submit your data alongside the paper that uses it.

    Funny, many researchers would take the lesson as burn every extant
    copy of your source data after publication. If you must keep the data,
    never let anybody see it.
    Of course this is official practice in
    pharmaceutical research, and soon to become so in economics.

    The lesson for the rest of us is: don’t accept a paper for publication without
    a copy of all the raw data already placed on escrow, with no right by the
    author, once accepted, to alter or withdraw it.

    In economics, we have the further lesson: do not act on conclusions from
    papers reviewed only by freshwater economists.

  5. Mike Taylor Says:

    That is a distressingly cynical perspective, Nathan; all the more so because of the suspicion that you might well be right.

  6. Lesson number -1: understand what your data means, and thus what accuracy you can ascribe to it.

    i.e.: do not give me 0.1% values for something that is 12 mm long and that you measured with a digital caliper. 0.1% of it would be 0.012 mm, way below the accuracy of your tool.

  7. […] as the string of failed replications in oncology and in social psychology is showing. See also the trivial but crucial spreadsheet error in the economics paper that underlies many austerity […]

  8. […] as the string of failed replications in oncology and in social psychology is showing. See also the trivial but crucial spreadsheet error in the economics paper that underlies many austerity […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: