Sauropod Vertebra Picture of the Week

The REF (Research Excellence Framework) is a time-consuming exercise that UK universities have to go through every few years to assess and demonstrate the value of their research to the government; the way funding is allocated between universities is largely dependent on the results of the REF. The exercise is widely resented, in part because the processes of preparing and reviewing the submissions are so time-consuming.

Dorothy Bishop has noted that results of REF assessments correlate strongly with departmental H-indexes (and suggested that we could save on the cost of future REFs by just using that H-index instead of the time-consuming peer-review process).

But it’s also been shown that H-index is strongly correlated with the simple number of publications. A seductive but naive conclusion would be: “we could just count publications for the next REF!”

But of course if we simply allocated research funding across universities on the basis of how many papers they produce, they would — they would have to — respond by simply racing to write more and more papers. Our already overloaded ability to assimilate new information would be further flooded. It’s a perverse incentive.

So this is a classic example of a very important general dictum: measure the thing you’re actually interested in, not a proxy. I don’t know if this dictum has been given a name yet, but it ought to be.

Measuring the thing we’re interested in is often difficult and time-consuming. But since only the thing we measure will ever be optimised for, we must measure the thing we want optimised — in this case, quality of research rather than quantity. That would still be true even if the correlation between REF assessments and departmental H-indexes was absolutely perfect. Because that correlation is an accident (in the philosophical sense); changing the circumstances will break that correlation.

No doubt all this reasoning is very familiar and painfully basic to people who have been working on the problem of metrics in assessment for years; to them, I apologise. For everyone else, I hope this comment provides some grains of insight.

[I originally wrote the initial form of this post as a comment on the Royal Society blog how should research and impact be assessed?, but I’m still waiting for it to be approved, so here it is.]