Correlation of impact factor and citations: a personal case-study

December 13, 2013

It’s now widely understood among researchers that the impact factor (IF) is a statistically illiterate measure of the quality of a paper. Unfortunately, it’s not yet universally understood among administrators, who in many places continue to judge authors on the impact factors of the journals they publish in. They presumably do this on the assumption that impact factor is a proxy for, or predictor of, citation count, which is turn is assumed to correlate with influence.

As shown by Lozano et al. (2012), the correlation between IF and citations is in fact very weak — r2 is about 0.2 — and has been progressively weakening since the dawn of the Internet era and the consequent decoupling of papers from the physical journal that they appear in. This is a counter-intuitive finding: given that the impact factor is calculated from citation counts you’d expect it to correlate much more strongly. But the enormous skew of citation rates towards a few big winners renders the average used by the IF meaningless.

To bring this home, I plotted my own personal impact-factor/citation-count graph. I used Google Scholar’s citation counts of my articles, which recognises 17 of my papers; then I looked up the impact factors of the venues they appeared in, plotted citation count against impact factor, and calculated a best-fit line through my data-points. Here’s the result (taken from a slide in my Berlin 11 satellite conference talk):


I was delighted to see that the regression slope is actually negative: in my case at least, the higher the impact factor of the venue I publish in, the fewer citations I get.

There are a few things worth unpacking on that graph.

First, note the proud cluster on the left margin: publications in venues with impact factor zero (i.e. no impact factor at all). These include papers in new journals like PeerJ, in perfectly respectable established journals like PaleoBios, edited-volume chapters, papers in conference proceedings, and an arXiv preprint.

My most-cited paper, by some distance, is Head and neck posture in sauropod dinosaurs inferred from extant animals (Taylor et al. 2009, a collaboration between all three SV-POW!sketeers). That appeared in Acta Palaeontologia Polonica, a very well-respected journal in the palaeontology community but which has a modest impact factor of 1.58.

My next most-cited paper, the Brachiosaurus revision (Taylor 2009), is in the Journal of Vertebrate Palaeontology — unquestionably the flagship journal of our discipline, despite its also unspectacular impact factor of 2.21. (For what it’s worth, I seem to recall it was about half that when my paper came out.)

In fact, none of my publications have appeared in venues with an impact factor greater than 2.21, with one trifling exception. That is what Andy Farke, Matt and I ironically refer to as our Nature monograph (Farke et al. 2009). It’s a 250-word letter to the editor on the subject of the Open Dinosaur Project. (It’ a subject that we now find profoundly embarrassing given how dreadfully slowly the project has progressed.)

Google Scholar says that our Nature note has been cited just once. But the truth is even better: that one citation is in fact from an in-prep manuscript that Google has dug up prematurely — one that we ourselves put on Google Docs, as part of the slooow progress of the Open Dinosaur Project. Remove that, and our Nature note has been cited exactly zero times. I am very proud of that record, and will try to preserve it by persuading Andy and Matt to remove the citation from the in-prep paper before we submit. (And please, folks: don’t spoil my record by citing it in your own work!)

What does all this mean? Admittedly, not much. It’s anecdote rather than data, and I’m posting it more because it amuses me than because it’s particularly persuasive. In fact if you remove the anomalous data point that is our Nature monograph, the slope becomes positive — although it’s basically meaningless, given that all my publications cluster in the 0–2.21 range. But then that’s the point: pretty much any data based on impact factors is meaningless.



10 Responses to “Correlation of impact factor and citations: a personal case-study”

  1. Samuel Says:

    The impact factor off a journal, in fact any journal level metric, should be understood as a measure of the quality of the journals processes. That is, it is a measure of how effective the journal is at choosing its editorial board and its peer reviewers, and how effectively the journal’s workflow process allows them to work together to collectively assess the quality of papers.

    You would not expect any individual paper to necessarily have high citations because it is quite possible for a paper to be high quality but not garner high citations for essentially accidental reasons – perhaps, for example, it is ahead of its time, or in an area that is not currently attracting a lot of attention.

    A paper in Nature, for example, had a lot of very highly qualified people look at it and say collectively that they thought it was a paper worthy of being in Nature. We judge the quality of their assessment by knowing that they have a track record of identifying important papers – their journal has a high IF – however this does not mean that every single paper they identify is going to have higher citations, or that they are failing if some papers are not cited at all.

    By the way, the above argument is to clarify the value of journal level metrics – I am not say that the IF is the best possible journal level metric, indeed there are any number of valid criticisms you could make of the way in which it is calculated, most importantly I think it is too sensitive to outliers. But that is an aside, the point is that journal level metrics have real value, something to consider given how often article level metrics are held out as a holy grail.

  2. Mike Taylor Says:

    Thanks for this, Samuel. But no:

    The impact factor off a journal, in fact any journal level metric, should be understood as a measure of the quality of the journals processes.

    The impact factor of a journal can only be understood as a measure of what the impact factor of the journal is. That’s all. It’s an arbitary, irreproducible, negotiable number that is subject to gaming both outright and more subtle. For a breakdown, see PLOS Medicine’s classic paper The Impact Factor Game — in particular this part:

    During the course of our discussions with Thompson Scientific, PLoS Medicine’s potential impact factor — based on the same articles published in the same year — seesawed between as much as 11 (when only research articles are entered into the denominator) to less than 3 (when almost all article types in the magazine section are included, as Thomson Scientific had initially done—wrongly, we argued, when comparing such article types with comparable ones published by other medical journals).

    Your contention that “a paper in Nature, for example, had a lot of very highly qualified people look at it and say collectively that they thought it was a paper worthy of being in Nature” comes at a rather delightful time, an hour or so after my attention was drawn to this graph in a Nature article published in June this year:

    It plots the probability P against not-P (in other words, x against 1-x) and — amazingly! — gets a straight line. The editorial and review process that allowed this through (not to mention the authorial process) is laughable.

  3. 220mya Says:

    You don’t account for time though – the longer a paper has been published, the more time it has to accrue citations. So one should ideally be normalizing for this.

  4. Matt Wedel Says:

    I’d like to actually see the graph without the Nature thing, and with the citations normalized for time since publication as Randy suggested. I mean, as long as you’re going to the trouble of making a graph, it might as well be maximally informative. It would be interesting to see if there is a correlation among the low-IF journals where most of us live most of the time.

  5. Richard Butler Says:

    I had this data to hand for my publications for various reasons, and calculated it for my 2005–2012 papers with citations normalised by numbers of years since publication. I excluded a Nature paper as an outlier (three times higher than any other impact factor). The result? For me, there is indeed a significant positive correlation between impact factor and citations (Pearson’s r = 0.50, p = 0.0002, n = 51). Make of that what you will, but the correlation exists, even if it only explains a relatively small proportion of the variance.

  6. Mike Taylor Says:

    Interesting. Seems your result is signficant but weak — r = 0.50 suggests a lot of scatter. It would be interesting to see your plot — do you have it available?

  7. Mike Taylor Says:

    I’m posting this on Richard Butler’s behalf, with his permission — Mike.

    Here is the plot. I don’t know how to insert it into a blog comment. There is quite a lot of scatter – notably: three highly cited papers in conference volumes without impact factors (Geological Society London Special Publications, Zitteliana B); my 2008 ornithischian phylogeny paper which is my best cited paper but in a relatively low impact journal (Journal of Systematic Palaeontology); a Biological Reviews paper from 2009 that hasn’t been well cited. If you exclude these five papers then r2 jumps to 0.62 based on 46 publications.

    Impact fact vs. citation-count correlation for Richard Butler

    Note that the underlying data was put together a couple of years ago – I did update citations, but not impact factors (more work), so the impact factors are a year or two old. That shouldn’t affect the general pattern though.

    My conclusion: there is scatter and anomalies, but overall there is an underlying relationship between impact factor and citations. It might not come through in your data because if we ignore the Nature letter, there’s not much range in the impact factors for the journals you’ve published in.

    I am not defending IF or generalising this result in any way, merely noting that the correlation exists in my data.

  8. David Marjanović Says:

    A high impact factor often tells you that a journal isn’t very specialized, but read by more people. This should mean that more people know about any paper in that journal and end up citing it.

    It holds up for my publications, but they’re so few that I haven’t bothered making a graph, and the paper with by far the highest number of citations and by far the highest IF (Systematic Biology, varies from 7 to 10 between years) is also my first paper. Indeed, except for the reversal between the two 2008 papers, each of my papers has more citations than any that is younger than it.* The first paper has, however, been cited by molecular biologists as well as paleontologists…

    * The 2010 “publication” that has been cited once is my doctoral thesis, which consists of my first three papers, an early draft of my 2013a paper, and an early draft of a manuscript I’m still working on; the one citation, BTW, is by the 2013a paper. – The five uncited “publications” at the bottom are conference abstracts.

  9. Heteromeles Says:

    Cool! A dust bunny distribution outside of vegetation ecology. It’s so called because most of the points are piled in the corner where the axes meet. These graphs are very, very hard to normalize, incidentally.

  10. […] before you sign the copyright transfer, the inability of impact factor to predict citation count (post to come), the childishness of evaluating individuals by journal rank, and the knotty problem of who should […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: