March 17, 2017
I’ve been on Twitter since April 2011 — nearly six years. A few weeks ago, for the first time, something I tweeted broke the thousand-retweets barrier. And I am really unhappy about it. For two reasons.
First, it’s not my own content — it’s a screen-shot of Table 1 from Edwards and Roy (2017):
And second, it’s so darned depressing.
The problem is a well-known one, and indeed one we have discussed here before: as soon as you try to measure how well people are doing, they will switch to optimising for whatever you’re measuring, rather than putting their best efforts into actually doing good work.
In fact, this phenomenon is so very well known and understood that it’s been given at least three different names by different people:
- Goodhart’s Law is most succinct: “When a measure becomes a target, it ceases to be a good measure.”
- Campbell’s Law is the most explicit: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
- The Cobra Effect refers to the way that measures taken to improve a situation can directly make it worse.
As I say, this is well known. There’s even a term for it in social theory: reflexivity. And yet we persist in doing idiot things that can only possibly have this result:
- Assessing school-teachers on the improvement their kids show in tests between the start and end of the year (which obviously results in their doing all they can depress the start-of-year tests).
- Assessing researchers by the number of their papers (which can only result in slicing into minimal publishable units).
- Assessing them — heaven help us — on the impact factors of the journals their papers appear in (which feeds the brand-name fetish that is crippling scholarly communication).
- Assessing researchers on whether their experiments are “successful”, i.e. whether they find statistically significant results (which inevitably results in p-hacking and HARKing).
What’s the solution, then?
I’ve been reading the excellent blog of economist Tim Harford, for a while. That arose from reading his even more excellent book The Undercover Economist (Harford 2007), which gave me a crash-course in the basics of how economies work, how markets help, how they can go wrong, and much more. I really can’t say enough good things about this book: it’s one of those that I feel everyone should read, because the issues are so important and pervasive, and Harford’s explanations are so clear.
In a recent post, Why central bankers shouldn’t have skin in the game, he makes this point:
The basic principle for any incentive scheme is this: can you measure everything that matters? If you can’t, then high-powered financial incentives will simply produce short-sightedness, narrow-mindedness or outright fraud. If a job is complex, multifaceted and involves subtle trade-offs, the best approach is to hire good people, pay them the going rate and tell them to do the job to the best of their ability.
I think that last part is pretty much how academia used to be run a few decades ago. Now I don’t want to get all misty-eyed and rose-tinted and nostalgic — especially since I wasn’t even involved in academia back then, and don’t know from experience what it was like. But could it be … could it possibly be … that the best way to get good research and publications out of scholars is to hire good people, pay them the going rate and tell them to do the job to the best of their ability?
[Read on to Why do we manage academia so badly?]
- Edwards, Marc A., and Siddhartha Roy. 2017. Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition. Environmental Engineering Science 34(1):51-61.
- Harford, Tim. 2007. The Undercover Economist. Abacus (Little, Brown). 384 pages. [Amazon US, Amazon UK]
Here is a nicely formatted full-page version of the Edwards and Roy table, for you to print out and stick on all the walls of your university. My thanks to David Roberts for preparing it.
September 18, 2016
I have before me the reviews for a submission of mine, and the handling editor has provided an additional stipulation:
Authority and date should be provided for each species-level taxon at first mention. Please ensure that the nominal authority is also included in the reference list.
In other words, the first time I mention Diplodocus, I should say “Diplodocus Marsh 1878″; and I should add the corresponding reference to my bibliography.
What do we think about this?
I used to do this religiously in my early papers, just because it was the done thing. But then I started to think about it. To my mind, it used to make a certain amount of sense 30 years ago. But surely in 2016, if anyone wants to know about the taxonomic history of Diplodocus, they’re going to go straight to Wikipedia?
I’m also not sure what the value is in providing the minimal taxonomic-authority information rather then, say, morphological information. Anyone who wants to know what Diplodocus is would be much better to go to Hatcher 1901, so wouldn’t we serve readers better if we referred to “Diplodocus (Hatcher 1901)”
Now that I come to think of it, I included “Giving the taxonomic authority after first use of each formal name” in my list of
Idiot things that we we do in our papers out of sheer habit three and a half years ago.
Should I just shrug and do this pointless busywork to satisfy the handling editor? Or should I simply refuse to waste my time adding information that will be of no use to anyone?
- Hatcher, Jonathan B. 1901. Diplodocus (Marsh): its osteology, taxonomy and probable habits, with a restoration of the skeleton. Memoirs of the Carnegie Museum 1:1-63 and plates I-XIII.
- Marsh, O. C. 1878. Principal characters of American Jurassic dinosaurs, Part I. American Journal of Science, series 3 16:411-416.
April 13, 2016
the whole idea is idiotic.
Plagiarism is “presenting someone else’s work or ideas as your own“. So self-plagiarism is presenting your own work or ideas as your own. Which is nonsense.
Can we please abandon this unhelpful and misleading phrase?
Note added subsequently, in response to Snarky Mᶜ̵Snarkface’s tweet: my real point is that discussion of the practice is actively confused by the use of this misleading term for it.
March 7, 2016
This one is for journalists and other popularizers of science. I see a lot of people writing that “scientists believe” this or that, when talking about hadrons or hadrosaurs or other phenomena grounded in evidence.
Pet peeve: believing is what people do in the absence of evidence, or despite evidence. Scientists often have to infer, estimate, and even speculate, but all of those activities are grounded in evidence and reason, not belief.1
In addition to doing science, scientists may also believe in the proper, spiritual sense, in which case you are free to explain what certain individual scientists believe. But that’s not how the word “believe” is used most of the time when it comes up in science stories.
So stop it. It’s lazy, and it’s damaging, because it gives (some) people the impression that scientists are clueless buffoons who make stuff up out of the whole cloth in a cynical bid to keep their jobs. Given that we have an entire political party pushing that view and trying to defund science and education at every turn, we don’t need that caricature promoted any further.
Even if you don’t accept that argument, it’s still bad writing. Good writing explains why people think as they do. So do that instead. In addition to the aforementioned “infer”, “estimate”, and “speculate”, you can use “surmise”, “reason”, “predict”, or – if you must – “think”. “Scientists have found” would be better still.
Best of all would be if the “scientists X” clause was preceded by, “Based on this evidence” (which you’ve just explained in the previous sentences), so readers can connect cause (evidence) and effect (scientists think) – which is what science is mostly about in the first place.
1. I realize that I am grossly oversimplifying – evidence, reason, and belief can interact in complicated ways in both spiritual and scientific spheres. But my purpose here is fixing poor word choice, not exploring that interaction.
January 28, 2016
Stand by . . . grumpy old man routine compiling . . .
So, someone at Sony decided that an Angry Birds movie would be a good idea, about three years after the Angry Birds “having a moment” moment was over. There’s a trailer for it now, and at the end of the trailer, a bird pees for like 17 seconds (which is about 1/7 of my personal record, but whatever).
And now I see these Poindexters all over the internet pushing their glasses up their noses and typing, “But everyone knows that birds don’t pee! They make uric acid instead! That’s the white stuff in ‘bird poop’. Dur-hur-hur-hurrr!” I am reasonably sure these are the same people who harped on the “inaccuracy” of the peeing Postosuchus in Walking With Dinosaurs two decades ago. (Honestly, how I didn’t get this written and posted in our first year of blogging is quite beyond my capacity.)
Congratulations, IFLScientists, on knowing One Fact about nature. Tragically for you, nature knows countless facts, and among them are that birds and crocodilians can pee. And since extant dinosaurs can and do pee, extinct ones probably could as well.
Now, it is true that crocs (mostly) and birds (always?) release more of their nitrogenous waste as uric acid than as urea. But their bodies produce both compounds. So does yours. We mammals are just shifted waaaay more heavily toward urea than uric acid, and extant archosaurs – and many (but not all) other reptiles to boot – are shifted waaaay more heavily toward uric acid than urea. Alligators also make a crapload of ammonia, but that’s a story for another time.
BUT, crucially, birds and crocs almost always release some clear, watery, urea-containing fluid when they dump the whitish uric acid, as shown in this helpful diagram that I stole from International Cockatiel Resource:
If you’ve never seen this, you’re just not getting to the bird poop fast enough – the urine is drying up before you notice it. Pick up the pace!
Sometimes birds and crocs save up a large quantity of fluid, and then flush everything out of their cloacas and lower intestines in one shot, as shown in the photos dribbled through this post. Which has led to some erroneous reports that ostriches have urinary bladders. They don’t, they just back up lots of urine into their colons. Many birds recapture some water and minerals that way, and thereby concentrate their wastes and save water – basically using the colon as a sort of second-stage kidney (Skadhauge 1976).
[UPDATE the next day: To be perfectly clear, all that’s going on here is that the birds and crocs keep their cloacal sphincters closed. The kidneys keep on producing urine and uric acid, and with no way out (closed sphincter) and nowhere else to go (no bladder – although urinary bladders have evolved repeatedly in lizards), the pee backs up into the colon. So if you’re wondering if extinct dinosaurs needed some kind of special adaptation to be able to pee, the answer is no. Peeing is an inherent possibility, and in fact the default setting, for any reptile that can keep its cloaca shut.]
Aaaanyway, all those white urate solids tend to make bird pee more whitish than yellow, as shown in the photos. I have seen a photo of an ostrich making a good solid stream from cloaca to ground that was yellow, but that was years ago and frustratingly I haven’t been able to relocate it. Crocodilians seem to have no problem making a clear, yellowish pee-stream, as you can see in many hilarious YouTube videos of gators peeing on herpetologists and reporters, which I am putting at the bottom of this post so as not to break up the flow of the rant.
You can explore this “secret history” of archosaur pee by entering the appropriate search terms into Google Scholar, where you’ll find papers with titles like:
- “Technique for the collection of clear urine from the Nile crocodile (Crocodylus niloticus)” (Myburgh et al. 2012)
- “Movement of urine in the lower colon and cloaca of ostriches” (Duke et al. 1995)
- “Plasma homeostasis and cloacal urine composition in Crocodylus porosus caught along a salinity gradient” (Grigg 1981)
- “Cloacal absorption of urine in birds” (Skadhauge 1976)
- “The cloacal storage of urine in the rooster” (Skadhauge 1968)
I’ve helpfully highlighted the operative term, to reinforce the main point of the post. Many of these papers are freely available – get the links from the References section below. A few are paywalled – really, Elsevier? $31.50 for a half-century-old paper on chicken pee? – but I’m saving them up, and I’ll be happy to lend a hand to other scholars who want to follow this stream of inquiry. If you’re really into the physiology of birds pooling pee in their poopers, the work of Erik Skadhauge will be a gold mine.
Now, to be fair, I seriously doubt that any bird has ever peed for 17 seconds. But the misinformation abroad on the net seems to be more about whether birds and other archosaurs can pee at all, rather than whether a normal amount of bird pee was exaggerated for comedic effect in the Angry Birds trailer.
In conclusion, birds and crocs can pee. Go tell the world.
And now, those gator peeing videos I promised:
Jan. 30, 2016: I just became aware that I had missed one of the best previous discussions of this topic, with one of the best videos, and the most relevant citations! The post is this one, by Brian Switek, which went up almost two years ago, the video is this excellent shot of an ostrich urinating and then defecating immediately after:
…and the citations are McCarville and Bishop (2002) – an SVP poster about a possible sauropod pee-scour, which is knew about but didn’t mention yet because I was saving it for a post of its own – and Fernandes et al. (2004) on some very convincing trace fossils of dinosaurs peeing on sand, from the Lower Cretaceous of Brazil. In addition to being cogent and well-illustrated, the Fernandes et al. paper has the lovely attribute of being freely available, here.
So, sorry, Brian, that I’d missed your post!
And for everyone else, stand by for another dinosaur pee post soon. And here’s one more video of an ostrich urinating (not pooping as the video title implies). The main event starts about 45 seconds in.
- Duke, G.E., Degen, A.A. and Reynhout, J.K., 1995. Movement of urine in the lower colon and cloaca of ostriches. Condor, 97, pp.165-173.
- Fernandes, M., Fernandes, L., Souto, P. 2004. Occurrence of urolites related to dinosaurs in the Lower Cretaceous of the Botucatu Formation, Paraná basin, São Paulo State, Brazil. Revista Brasileira de Paleontologia. 7(2), pp.263-268.
- Grigg, G.C., 1981. Plasma homeostasis and cloacal urine composition in Crocodylus porosus caught along a salinity gradient. Journal of Comparative Physiology, 144(2), pp.261-270.
- McCarville, K., Bishop, G. 2002. To pee or not to pee: evidence for liquid urination in sauropod dinosaurs. In: 62nd Annual Meeting of the Society of Vertebrate Paleontology, Abstract Book. Journal of Vertebrate Paleontology 22(3, Supplement), p. 85A.
- Myburgh, J.G., Huchzermeyer, F.W., Soley, J.T., Booyse, D.G., Groenewald, H.B., Bekker, C., Iguchi, T. and Guillette, L.J., 2012. Technique for the collection of clear urine from the Nile crocodile (Crocodylus niloticus). Journal of the South African Veterinary Association, 83(1), pp.1-7.
- Skadhauge, E., 1968. The cloacal storage of urine in the rooster. Comparative Biochemistry and Physiology, 24(1), pp.7-18.
- Skadhauge, E., 1976. Cloacal absorption of urine in birds. Comparative Biochemistry and Physiology Part A: Physiology, 55(2), pp.93-98.
January 25, 2016
Like Stephen Curry, we at SV-POW! are sick of impact factors. That’s not news. Everyone now knows what a total disaster they are: how they are signficantly correlated with retraction rate but not with citation count; how they are higher for journals whose studies are less statistically powerful; how they incentivise bad behaviour including p-hacking and over-hyping. (Anyone who didn’t know all that is invited to read Brembs et al.’s 2013 paper Deep impact: unintended consequences of journal rank, and weep.)
Its 2016. Everyone who’s been paying attention knows that impact factor is a terrible, terrible metric for the quality of a journal, a worse one for the quality of a paper, and not even in the park as a metric for the quality of a researcher.
Unfortunately, “everyone who’s been paying attention” doesn’t seem to include such figures as search committees picking people for jobs, department heads overseeing promotion, tenure committees deciding on researchers’ job security, and I guess granting bodies. In the comments on this blog, we’ve been told time and time and time again — by people who we like and respect — that, however much we wish it weren’t so, scientists do need to publish in high-IF journals for their careers.
What to do?
It’s a complex problem, not well suited to discussion on Twitter. Here’s what I wrote about it recently:
The most striking aspect of the recent series of Royal Society meetings on the Future of Scholarly Scientific Communication was that almost every discussion returned to the same core issue: how researchers are evaluated for the purposes of recruitment, promotion, tenure and grants. Every problem that was discussed – the disproportionate influence of brand-name journals, failure to move to more efficient models of peer-review, sensationalism of reporting, lack of replicability, under-population of data repositories, prevalence of fraud – was traced back to the issue of how we assess works and their authors.
It is no exaggeration to say that improving assessment is literally the most important challenge facing academia.
This is from the introduction to a new paper which came out today: Taylor (2016), Better ways to evaluate research and researchers. In eight short pages — six, really, if you ignore the appendix — I try to get to grips with the historical background that got us to where we are, I discuss some of the many dimensions we should be using to evaluate research and researchers, and I propose a family of what I call Less Wrong Metrics — LWMs — that administrators could use if they really absolutely have to put a single number of things.
(I was solicited to write this by SPARC Europe, I think in large part because of things I have written around this subject here on SV-POW! My thanks to them: this paper becomes part of their Briefing Papers series.)
Next time I’ll talk about the LWM and how to calculate it. Those of you who are impatient might want to read the actual paper first!
November 10, 2015
One thing that always bemuses me is the near-absolute serendipity of the academic job market. To get into research careers takes at least a decade of very deliberate, directed work, and then at the end you basically toss your diploma into a whirlwind and see where it lands. After all of that careful planning, almost all of us end up where we do based on the random (to us) set of jobs available in the narrow window in which we’re searching.
Did you dream of being curator at Museum X, or professor at University Y? Well, tough, those jobs went to Dr. Graduated-Two-Years-Sooner and Lucky Nature Paper, PhD, and they’re not retiring for three or four decades. Or maybe your dream job comes open right after you, your spouse, and your kids get settled in at your new acceptable-but-not-quite-dream job. Uproot or stay the course? Or what would be your dream job finally comes open but they’re looking for new junior faculty and you just got tenure at Tolerable State U.
This drastic mismatch between carefulness of preparation and randomness of outcome was present even pre-2008. The craptastic academic job market since then has only whetted the central irony’s keen edge. Getting grants and getting jobs is now basically a lottery. I’m not saying that good jobs don’t go to good people – they almost always do – but there are a lot of good people in jobs they never imagined having. And, sadly, plenty of good people who are now working outside of the field they prepared for because of the vicissitudes of the job market. A handful of years sooner or later and they might be sitting pretty.
This is on my mind because I recently had lunch with a physician friend from work and he was talking about applying for jobs as a doctor. “The first thing everyone tells you,” he said, “is decide what part of the country you want to live in first, then apply for the jobs that are there.” Doctors can do that because there are more than 800,000 of them active in the US. Paleontologists are mighty rarified by comparison – it’s hard to say how many of us there are, but probably not more than 2000 active in vert paleo. So the usual advice for budding biologists and paleontologists is exactly opposite that for physicians: “Forget about living where you want. Go wherever the job is and make the best of it.”
Oddly enough, I don’t remember this ever coming up in grad school. It’s something Vicki and I figured out at the end, as we started the process of applying for positions. There are alternate universes where we are at Marshall (they offered us both jobs, but not as attractive as UC Merced at the time), or at Northern Arizona (which is bittersweet because we have totally fallen in love with Flagstaff just in the past three years), or other places. If I were choosing a job site based on everything other than the institution, I’d spring for somewhere in Arizona or the intermountain west in a heartbeat.
But with all that said, we are happy here. It’s funny, when we got the job offers down here I thought, “LA? Crap, there goes the outdoor part of my life.” But Claremont has lots of parks, it’s tucked up against the San Gabriels and I can get into the mountains in 30 minutes, or out to the desert in 90. I’m spending more time outdoors than I have since I was a kid growing up in rural Oklahoma.
So I’m not complaining about my personal situation. Vicki and I both landed on our feet – and the fact that we both managed to stick the landing at the same institution is little short of miraculous. But we still had to step into the job market hurricane to get here.
If you’re a grad student and you’re reading this, I didn’t write it to freak you out. Just to let you know that it’s coming, and there are things you can do to improve your chances. Be aggressively curious. Write. Publish. Give good talks (and give lots of talks so you can become good at it). Broaden your skill set – if you’re going into paleo, knowing how to teach human anatomy probably doubles or triples the number of available jobs at any one time, even if many of them are not the jobs you’ve been dreaming of.
Then, at the end, pour yourself one stiff drink and cast your fortune to the winds.