Correlation of impact factor and citations: a personal case-study
December 13, 2013
It’s now widely understood among researchers that the impact factor (IF) is a statistically illiterate measure of the quality of a paper. Unfortunately, it’s not yet universally understood among administrators, who in many places continue to judge authors on the impact factors of the journals they publish in. They presumably do this on the assumption that impact factor is a proxy for, or predictor of, citation count, which is turn is assumed to correlate with influence.
As shown by Lozano et al. (2012), the correlation between IF and citations is in fact very weak — r2 is about 0.2 — and has been progressively weakening since the dawn of the Internet era and the consequent decoupling of papers from the physical journal that they appear in. This is a counter-intuitive finding: given that the impact factor is calculated from citation counts you’d expect it to correlate much more strongly. But the enormous skew of citation rates towards a few big winners renders the average used by the IF meaningless.
To bring this home, I plotted my own personal impact-factor/citation-count graph. I used Google Scholar’s citation counts of my articles, which recognises 17 of my papers; then I looked up the impact factors of the venues they appeared in, plotted citation count against impact factor, and calculated a best-fit line through my data-points. Here’s the result (taken from a slide in my Berlin 11 satellite conference talk):
I was delighted to see that the regression slope is actually negative: in my case at least, the higher the impact factor of the venue I publish in, the fewer citations I get.
There are a few things worth unpacking on that graph.
First, note the proud cluster on the left margin: publications in venues with impact factor zero (i.e. no impact factor at all). These include papers in new journals like PeerJ, in perfectly respectable established journals like PaleoBios, edited-volume chapters, papers in conference proceedings, and an arXiv preprint.
My most-cited paper, by some distance, is Head and neck posture in sauropod dinosaurs inferred from extant animals (Taylor et al. 2009, a collaboration between all three SV-POW!sketeers). That appeared in Acta Palaeontologia Polonica, a very well-respected journal in the palaeontology community but which has a modest impact factor of 1.58.
My next most-cited paper, the Brachiosaurus revision (Taylor 2009), is in the Journal of Vertebrate Palaeontology — unquestionably the flagship journal of our discipline, despite its also unspectacular impact factor of 2.21. (For what it’s worth, I seem to recall it was about half that when my paper came out.)
In fact, none of my publications have appeared in venues with an impact factor greater than 2.21, with one trifling exception. That is what Andy Farke, Matt and I ironically refer to as our Nature monograph (Farke et al. 2009). It’s a 250-word letter to the editor on the subject of the Open Dinosaur Project. (It’ a subject that we now find profoundly embarrassing given how dreadfully slowly the project has progressed.)
Google Scholar says that our Nature note has been cited just once. But the truth is even better: that one citation is in fact from an in-prep manuscript that Google has dug up prematurely — one that we ourselves put on Google Docs, as part of the slooow progress of the Open Dinosaur Project. Remove that, and our Nature note has been cited exactly zero times. I am very proud of that record, and will try to preserve it by persuading Andy and Matt to remove the citation from the in-prep paper before we submit. (And please, folks: don’t spoil my record by citing it in your own work!)
What does all this mean? Admittedly, not much. It’s anecdote rather than data, and I’m posting it more because it amuses me than because it’s particularly persuasive. In fact if you remove the anomalous data point that is our Nature monograph, the slope becomes positive — although it’s basically meaningless, given that all my publications cluster in the 0–2.21 range. But then that’s the point: pretty much any data based on impact factors is meaningless.
References
- Farke, Andrew A., Michael P. Taylor and Mathew J. Wedel. 2009. Sharing: public databases combat mistrust and secrecy. Nature 461:1053.
- Lozano, George A., Vincent Larivière and Yves Gingras. 2012. The weakening relationship between the impact factor and papers’ citations in the digital age. Journal of the American Society for Information Science and Technology 63(11):2140-2145. doi:10.1002/asi.22731 [arXiv preprint]
- Taylor, Michael P. 2009. A re-evaluation of Brachiosaurus altithorax Riggs 1903 (Dinosauria, Sauropoda) and its generic separation from Giraffatitan brancai (Janensch 1914). Journal of Vertebrae Paleontology 29(3):787-806.
- Taylor, Michael P., Mathew J. Wedel and Darren Naish. 2009. Head and neck posture in sauropod dinosaurs inferred from extant animals. Acta Palaeontologica Polonica 54(2):213-230.
Sauropod Vertebra Picture of the Week 
December 13, 2013 at 8:18 pm
The impact factor off a journal, in fact any journal level metric, should be understood as a measure of the quality of the journals processes. That is, it is a measure of how effective the journal is at choosing its editorial board and its peer reviewers, and how effectively the journal’s workflow process allows them to work together to collectively assess the quality of papers.
You would not expect any individual paper to necessarily have high citations because it is quite possible for a paper to be high quality but not garner high citations for essentially accidental reasons – perhaps, for example, it is ahead of its time, or in an area that is not currently attracting a lot of attention.
A paper in Nature, for example, had a lot of very highly qualified people look at it and say collectively that they thought it was a paper worthy of being in Nature. We judge the quality of their assessment by knowing that they have a track record of identifying important papers – their journal has a high IF – however this does not mean that every single paper they identify is going to have higher citations, or that they are failing if some papers are not cited at all.
By the way, the above argument is to clarify the value of journal level metrics – I am not say that the IF is the best possible journal level metric, indeed there are any number of valid criticisms you could make of the way in which it is calculated, most importantly I think it is too sensitive to outliers. But that is an aside, the point is that journal level metrics have real value, something to consider given how often article level metrics are held out as a holy grail.
December 13, 2013 at 11:26 pm
Thanks for this, Samuel. But no:
The impact factor of a journal can only be understood as a measure of what the impact factor of the journal is. That’s all. It’s an arbitary, irreproducible, negotiable number that is subject to gaming both outright and more subtle. For a breakdown, see PLOS Medicine’s classic paper The Impact Factor Game — in particular this part:
Your contention that “a paper in Nature, for example, had a lot of very highly qualified people look at it and say collectively that they thought it was a paper worthy of being in Nature” comes at a rather delightful time, an hour or so after my attention was drawn to this graph in a Nature article published in June this year:
It plots the probability P against not-P (in other words, x against 1-x) and — amazingly! — gets a straight line. The editorial and review process that allowed this through (not to mention the authorial process) is laughable.
December 14, 2013 at 1:25 am
You don’t account for time though – the longer a paper has been published, the more time it has to accrue citations. So one should ideally be normalizing for this.
December 14, 2013 at 6:30 am
I’d like to actually see the graph without the Nature thing, and with the citations normalized for time since publication as Randy suggested. I mean, as long as you’re going to the trouble of making a graph, it might as well be maximally informative. It would be interesting to see if there is a correlation among the low-IF journals where most of us live most of the time.
December 16, 2013 at 8:57 am
I had this data to hand for my publications for various reasons, and calculated it for my 2005–2012 papers with citations normalised by numbers of years since publication. I excluded a Nature paper as an outlier (three times higher than any other impact factor). The result? For me, there is indeed a significant positive correlation between impact factor and citations (Pearson’s r = 0.50, p = 0.0002, n = 51). Make of that what you will, but the correlation exists, even if it only explains a relatively small proportion of the variance.
December 16, 2013 at 9:13 am
Interesting. Seems your result is signficant but weak — r = 0.50 suggests a lot of scatter. It would be interesting to see your plot — do you have it available?
December 16, 2013 at 11:54 am
I’m posting this on Richard Butler’s behalf, with his permission — Mike.
—
Here is the plot. I don’t know how to insert it into a blog comment. There is quite a lot of scatter – notably: three highly cited papers in conference volumes without impact factors (Geological Society London Special Publications, Zitteliana B); my 2008 ornithischian phylogeny paper which is my best cited paper but in a relatively low impact journal (Journal of Systematic Palaeontology); a Biological Reviews paper from 2009 that hasn’t been well cited. If you exclude these five papers then r2 jumps to 0.62 based on 46 publications.
Note that the underlying data was put together a couple of years ago – I did update citations, but not impact factors (more work), so the impact factors are a year or two old. That shouldn’t affect the general pattern though.
My conclusion: there is scatter and anomalies, but overall there is an underlying relationship between impact factor and citations. It might not come through in your data because if we ignore the Nature letter, there’s not much range in the impact factors for the journals you’ve published in.
I am not defending IF or generalising this result in any way, merely noting that the correlation exists in my data.
December 17, 2013 at 5:37 pm
A high impact factor often tells you that a journal isn’t very specialized, but read by more people. This should mean that more people know about any paper in that journal and end up citing it.
It holds up for my publications, but they’re so few that I haven’t bothered making a graph, and the paper with by far the highest number of citations and by far the highest IF (Systematic Biology, varies from 7 to 10 between years) is also my first paper. Indeed, except for the reversal between the two 2008 papers, each of my papers has more citations than any that is younger than it.* The first paper has, however, been cited by molecular biologists as well as paleontologists…
* The 2010 “publication” that has been cited once is my doctoral thesis, which consists of my first three papers, an early draft of my 2013a paper, and an early draft of a manuscript I’m still working on; the one citation, BTW, is by the 2013a paper. – The five uncited “publications” at the bottom are conference abstracts.
December 18, 2013 at 5:01 pm
Cool! A dust bunny distribution outside of vegetation ecology. It’s so called because most of the points are piled in the corner where the axes meet. These graphs are very, very hard to normalize, incidentally.
January 10, 2014 at 2:43 pm
[…] before you sign the copyright transfer, the inability of impact factor to predict citation count (post to come), the childishness of evaluating individuals by journal rank, and the knotty problem of who should […]