Yesterday we all trooped off to the Millennium Stadium in Cardiff for a Staff Away Day. We didn’t actually get to play on the pitch of course, which wasn’t even there, as it had been removed to reveal a vast expanse of soil. Instead we were installed in the “Dragon Suite” for a discussion about our preparation for the forthcoming Research Excellence Framework.
Obviously I can’t post anything about our internal deliberations, but I’m sure departments up and down the United Kingdom are doing similar things so I thought I’d mention a few things which are already in the public domain and my personal reactions to them. I should also say that the opinions I express below are my own and not necessarily those of anyone else at Cardiff.
The first thing is the scale of the task facing members of the panel undertaking this assessment. Each research active member of staff is requested to submit four research publications (“outputs”) to the panel, and we are told that each of these will be read by at least two panel members. The panel comprises 20 members.
As a rough guess I’d say that the UK has about 40 Physics departments, and the average number of research-active staff in each is probably about 40. That gives about 1600 individuals for the REF. Actually the number of category A staff submitted to the 2008 RAE was 1,685.57 FTE (Full-Time Equivalent), pretty close to this figure. At 4 outputs per person that gives 6400 papers to be read. We’re told that each will be read by at least two members of the panel, so that gives an overall job size of 12800 paper-readings. There are 20 members of the panel, so that means that between 29th November 2013 (the deadline for submissions) and the announcement of the results in December 2014 each member of the panel will have to have read 640 research papers. That’s an average of about two a day…
Incidentally, as I’ve mentioned before, the Physics REF panel includes representatives from institutions in England, Scotland and Northern Ireland, but not Wales. The decision to exclude representation from Welsh physics departments was a disgrace, in my view.
Now we are told the panel will use their expert judgment to decide which outputs belong to the following categories:
- 4* World Leading
- 3* Internationally Excellent
- 2* Internationally Recognized
- 1* Nationally Recognized
- U Unclassified
There is an expectation that the so-called QR funding allocated as a result of the 2013 REF will be heavily weighted towards 4*, with perhaps a small allocation to 3* and probably nothing at all for lower grades. “Internationally recognized” research is probably worthless in the view of HEFCE, in other words. Will the papers belonging to the category “Not really understood by the panel member” suffer the same fate?
The panel members will apparently know enough about every single one of the papers they are going to read in order to place them into one of the above categories, especially the crucial ones “world-leading” or “internationally excellent”, both of which are obviously defined in a completely transparent and objective manner. Not.
We are told that after forming this judgement based on their expertise the panel members will “check” the citation information for the papers. This will be done using the SCOPUS service provided (no doubt at considerable cost) by Elsevier, which by sheer coincidence also happens to be a purveyor of ridiculously overpriced academic journals. I’ve just checked the citation information for some of my papers on SCOPUS, and found an alarming number of errors. No doubt Elsevier are on a nice little earner peddling meaningless data for the HECFE bean-counters, but I haven’t any confidence that it will add much value to the assessment process.
There have been high-profile statements to the effect that the REF will take no account of where the relevant “outputs” are published, including a recent pronouncement by David Willetts. On the face of it, that would suggest that a paper published in the spirit of Open Access in a free archive would not be disadvantaged. However, I very much doubt that will be the case.
I think if you look at the volume of work facing the REF panel members it’s pretty clear that citation statistics will be much more important for the Physics panel than we’ve been led to believe. The panel simply won’t have the time or the breadth of understanding to do an in-depth assessment of every paper, so will inevitably in many cases be led by bibliometric information. The fact that SCOPUS doesn’t cover the arXiv means that citation information will be entirely missing from papers just published there.
The involvement of a company like Elsevier in this system just demonstrates the extent to which the machinery of research assessment is driven by the academic publishing industry. The REF is now pretty much the only reason why we have to use traditional journals. It would be better for research, better for public accountability and better economically if we all published our research free of charge in open archives. It wouldn’t be good for academic publishing houses, however, so they’re naturally very keen to keep things just the way they are. The saddest thing is that we’re all so cowed by the system that we see no alternative but to participate in this scam.
Incidentally we were told before the 2008 Research Assessment Exercise that citation data would emphatically not be used; we were also told afterwards that citation data had been used by the Physics panel. That’s just one of the reasons why I’m very sceptical about the veracity of some of the pronouncements coming out from the REF establishment. Who knows what they actually do behind closed doors? All the documentation is shredded after the results are published. Who can trust such a system?
To put it bluntly, the apparatus of research assessment has done what most bureaucracies eventually do; it has become entirely self-serving. It is imposing increasingly ridiculous administrative burdens on researchers, inventing increasingly arbitrary assessment criteria and wasting increasing amounts of money on red tape which should actually be going to fund research.
And that’s all just about “outputs”. I haven’t even started on “impact”….
In the Dark 