I’ve been on Twitter since April 2011 — nearly six years. A few weeks ago, for the first time, something I tweeted broke the thousand-retweets barrier. And I am really unhappy about it. For two reasons.

First, it’s not my own content — it’s a screen-shot of Table 1 from Edwards and Roy (2017):

c49rdmlweaaa4if

And second, it’s so darned depressing.

The problem is a well-known one, and indeed one we have discussed here before: as soon as you try to measure how well people are doing, they will switch to optimising for whatever you’re measuring, rather than putting their best efforts into actually doing good work.

In fact, this phenomenon is so very well known and understood that it’s been given at least three different names by different people:

  • Goodhart’s Law is most succinct: “When a measure becomes a target, it ceases to be a good measure.”
  • Campbell’s Law is the most explicit: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
  • The Cobra Effect refers to the way that measures taken to improve a situation can directly make it worse.

As I say, this is well known. There’s even a term for it in social theory: reflexivity. And yet we persist in doing idiot things that can only possibly have this result:

  • Assessing school-teachers on the improvement their kids show in tests between the start and end of the year (which obviously results in their doing all they can depress the start-of-year tests).
  • Assessing researchers by the number of their papers (which can only result in slicing into minimal publishable units).
  • Assessing them — heaven help us — on the impact factors of the journals their papers appear in (which feeds the brand-name fetish that is crippling scholarly communication).
  • Assessing researchers on whether their experiments are “successful”, i.e. whether they find statistically significant results (which inevitably results in p-hacking and HARKing).

What’s the solution, then?

I’ve been reading the excellent blog of economist Tim Harford, for a while. That arose from reading his even more excellent book The Undercover Economist (Harford 2007), which gave me a crash-course in the basics of how economies work, how markets help, how they can go wrong, and much more. I really can’t say enough good things about this book: it’s one of those that I feel everyone should read, because the issues are so important and pervasive, and Harford’s explanations are so clear.

In a recent post, Why central bankers shouldn’t have skin in the game, he makes this point:

The basic principle for any incentive scheme is this: can you measure everything that matters? If you can’t, then high-powered financial incentives will simply produce short-sightedness, narrow-mindedness or outright fraud. If a job is complex, multifaceted and involves subtle trade-offs, the best approach is to hire good people, pay them the going rate and tell them to do the job to the best of their ability.

I think that last part is pretty much how academia used to be run a few decades ago. Now I don’t want to get all misty-eyed and rose-tinted and nostalgic — especially since I wasn’t even involved in academia back then, and don’t know from experience what it was like. But could it be … could it possibly be … that the best way to get good research and publications out of scholars is to hire good people, pay them the going rate and tell them to do the job to the best of their ability?

[Read on to Why do we manage academia so badly?]

References

Bonus

Here is a nicely formatted full-page version of the Edwards and Roy table, for you to print out and stick on all the walls of your university. My thanks to David Roberts for preparing it.

Many thanks to everyone who played pin-the-skull-on-the-carnivore. The answers are down at the bottom of this post, so if you’ve just arrived here and want to take the challenge, go here before you scroll down.

To fill up some space, let me point out how crazy variable the skulls of black bears, Ursus americanus, are.

My bear skull - left lateral reversed

Here’s the one I helped dig up, missing the occipital region. Note the double inflection in the dorsal outline that separates the forehead from both the snout and top of the head, and the way the nasal bones stick out at a very different angle from the maxilla.

Page Museum black bear skull

Here’s the skull of a black bear from the La Brea tar pits, in the Page Museum in L.A. I don’t know if this one was female or juvenile or what, but the dorsal margin of the skull is one mostly-smooth curve from occiput almost to incisors, with the nasals scarcely deviating at all. Lest you think these differences were caused by evolutionary change rather than intraspecific variation, similar “roundhead” bear skulls from modern times are here and here and near the bottom of this page.

It’s this variability that first got me thinking about doing the Carnivore Skull Challenge. I saw a couple of photos of skulls of wolverines, and except for having carnassial cheek teeth instead of flatter premolars and molars, the wolverine skulls look like they could fit right into the span of black bear skull variability (in shape; obviously they’re not nearly as big). Then I saw a hyena skull and thought that it wasn’t that far off from a wolverine either. A little more searching for plausible distractors and I was all set.

Here are the answers, by the way:

Carnivore skull challenge - answersIt’s kind of ironic, then, that the first two people to venture identifications picked out the black bear right away. In the very first comment, Dean got it almost all right except for swapping the seal and the fossa. Dean was also the first to get all of the skulls correctly identified, albeit on his second pass. Markus Bühler (of Cthulhu-sculpting fame) was the first to get them all the first time. Tom Nutter, our own Darren Naish, and microecos Neil also aced the test, although in light of the Page Museum bear skull shown above, I was amused to see Darren’s “D: Bear. Because forehead.” I guess it’s one of those presence-of-forehead-means-bear, absence-of-forehead-does-not-rule-out-bear things that logicians are always going on about.

I was really happy to see people getting the wolverine and hyena mixed up, because they really do look strikingly similar to me. It’s almost like hyena + bear = wolverine.

Brian Engh asked on Facebook when I was going to do one for sauropods. Patience, good sir! It’s on my to-do list.

Finally, speaking of bear skulls, you can get a sweet tiny bronze one with a hinged jaw as part of this already successful Kickstarter, or from Fireandbone.com once the Kickstarter ends.

Something very different, and very unexpected, tomorrow.

Carnivore skull challenge

In this image I have assembled photos of skulls (or casts of skulls) of six extant carnivores. I exclusively used photos from the Skulls Unlimited website because they had all the taxa I wanted, lit about the same and photographed from similar angles. The omission of scale indicators is deliberate.

Your mission, should you choose to accept it, is to match these skulls with the animals they came from. Here are their currently-understood hierarchical relationships, scientific names, and common names (aside: I know this is ugly, is there a way to make nested tables in WordPress?).

Carnivora

– – Herpestoidea

– – – – Eupleridae

– – – – – – Fossa, Cryptoprocta ferox

– – – – Hyaenidae

– – – – – – Brown hyena, Hyaena brunnea

– – Arctoidea

– – – – Ursoidea

– – – – – – American black bear, Ursus americanus

– – – – Musteloidea

– – – – – – European badger, Meles meles

– – – – – – Wolverine, Gulo gulo

– – – – Pinnipedia

– – – – – – Mediterranean monk seal,  Monachus monachus

If you accept the challenge, leave your guesses as comments below, but only if you’ve played fair–no checking websites, references, or your own skull collection! Don’t worry about being wrong, I freely admit that I would have flunked this bigtime if anyone else had inflicted it on me. I decided to set up this challenge after I noticed the striking similarity between two of these critters in particular; I’ll tell you which two when I post the reveal in a day or two.