Last week I read an interesting bit of news in the Times Higher that the forthcoming Research Excellence Framework (REF) seems to be getting cold feet about using citation numbers as a metric for quantifying research quality. I shouldn’t be surprised about that, because I’ve always thought it was very difficult to apply such statistics in a meaningful way. Nevertheless, I am surprised – because meaningfulness has never seemed to me to be very high on the agenda for the Research Excellence Framework….
There are many issues with the use of citation counts, some of which I’ve blogged about before, but I was interested to read another article in the Times Higher, in this weeks issue, commenting on the fact that some papers have ridiculously large author lists. The example picked by the author, Gavin Fairbairn (Professor of Ethics and Language at Leeds Metropolitan University), turns out – not entirely surprisingly – to be from the field of astronomy. In fact it’s The Sloan Digital Sky Survey: Technical Summary which is published in the Astronomical Journal and has 144 authors. It’s by no means the longest author list I’ve ever seen, in fact, but it’s certainly very long by the standards of the humanities. Professor Fairbairn goes on to argue, correctly, that there’s no way every individual listed among the authors could have played a part in the writing of the paper. On the other hand, the Sloan Digital Sky Survey is a vast undertaking and there’s no doubt that it required a large number of people to make it work. How else to give them credit for participating in the science than by having them as authors on the paper?
Long author lists are increasingly common in astronomy these days, not because of unethical CV-boosting but because so many projects involve large, frequently international, collaborations. The main problem from my point of view, however, is not the number of authors, but how credit is assigned for the work in exercises like the REF.
The basic idea about using citations is fairly sound: a paper which is important (or “excellent”, in REF language) will attract more citations than less important ones because more people will refer to it when they write papers of their own. So far, so good. However the total number of citations for even a very important paper depends on the size and publication rate of the community working in the field. Astronomy is not a particularly large branch of the physical sciences but is very active and publication rates are high, especially when it comes to observational work. In condensed matter physics citation rates are generally a lot lower, but that’s more to do with the experimental nature of the subject. It’s not easy, therefore, to compare from one field to another. Setting that issue to one side, however, we come to the really big issue, which is how to assign credit to authors.
You see, it’s not authors that get citations, it’s papers. Let’s accept that a piece of work might be excellent and that this excellence can be quantified by the number of citations N it attracts. Now consider a paper written by a single author that has excellence-measure N versus a paper with 100 authors that has the same number of citations. Don’t you agree that the individual author of the first paper must have generated more excellence than each of the authors of the second? It seems to me that it stands to reason that the correct way to apportion credit is to divide the number of citations by the number of authors (perhaps with some form of weighting to distinguish drastically unequal contributions). I contend that such a normalized citation count is the only way to quantify the excellence associated with an individual author.
Of course whenever I say this to observational astronomers they accuse me of pro-theory bias, because theorists tend to work in smaller groups than observers. However, that ignores the fact that not doing what I suggest leads to a monstrous overcounting of the total amout of excellence. The total amount of excellence spread around the community for the second paper in my example is not N but 100N. Hardly surprising, then, that observational astronomers tend to have such large h-indices – they’re all getting credit for each others contributions as well as their own! Most observational astronomers’ citation measures reduce by a factor of 3 or 4 when they’re counted properly.
I think of the citation game as being a bit like the National Lottery. Writing a paper is like buying a ticket. You can buy one yourself, or you can club together and buy one as part of a syndicate. If you win with your own ticket, you keep the whole jackpot. If a syndicate wins, though, you don’t expect each member to win the total amount – you have to share the pot between you.
In the Dark 