In the Dark

No sooner has the dust settled on the  2008 Research Assessment Exercise (RAE) when the Higher Education Funding Council for England (HEFCE) has tabled its proposals for a new system called the Research Excellence Framework (REF) in a 56-page consultation document that you can download and peruse at your leisure.

I won’t try to give a complete account of the new system except to say that apart from the change of acronym there won’t be much different. Many of us hoped that the new framework would involve a lighter touch than the RAE, so we could actually get on with research instead of filling in forms all our lives. Fat chance. You can call me cynical if you like, but I think it’s obvious that once you set up a monstrous bureaucratical nightmare like the RAE it is almost impossible to kill it off. Things like this gather their own momentum and become completely self-serving. The apparatus of research assessment no longer exists to fulfil a particular purpose. It exists because it exists.

It might be useful however to summarise the main changes:

1. The number of Units of Assessment and sub-panels is to be reduced from 67 to 30 and the number of main assessment panels from 15 to 4. This move is bound to prove controversial as it will clearly reduce the number of specialists involved in the quality appraisal side of things. However, the last RAE produced clear anomalies in the assessment carried out by different panels: physics overall did very poorly compared to other disciplines, for example. Having fewer panels might make it easier to calibrate different subjects. Might.
2. In REF the overall assessments are going to be based on three elements: research output (60%); impact (25%); and environment (15%). In the last RAE each panel was free to vary the relative contribution of different components to the overall score. Although the “research output” category is similar to the last RAE, it is now proposed to include citation measures in the overall assessment. Officially, that is. It’s an open secret that panel members did look at citations last time anyway.  Citation impact will however be used only for certain science and engineering subjects.  “Impact” is a new element and its introduction is  in line with the government’s agenda to pump research funds into things which will generate wealth, so this measure will probably shaft fundamental physics. “Environment” includes things like postgraduate numbers, research funding and the like; this is also similar to the RAE.
3. A roughly similar number of experts will be involved as in RAE 2008 – so it will be similarly expensive to run.
4. The consultation document asks whether the number of outputs submitted per person should be reduced from four to three, and also whether “substantive outputs” (whatever they are) should be “double-weighted”.
5. The results will be presented in terms of “profiles” as in 2008, with the percentage of activity at each level being given.
6. The consultation also suggests honing the description of “world-leading” (4*) and “internationally excellent” (3*) to achieve greater discrimination at the top end of the scale. This is deeply worrying, as well as completely absurd. The last RAE applied a steeply rising funding formula to the scores so that 4*:3*:2*:1* was weighted 7:3:1:0. However the fraction of  work in each category is subject to considerable uncertainty, amplified by the strong weighting.  If the categories are divided further then I can see an even steeper weighting emerging, with the likely outcome that small variations in the (subjective) assessment will lead to drastic variations in funding. Among the inevitable consequences of this will be that  some excellent research will lose out.

No doubt university administrators across the United Kingdom will already be plotting how best to play the new system. I think we need to remember, though, that deep cuts in public spending have been promised by both major political parties and there is a general election due next year. I can see the overall  budget for university research being slashed so we’ll be fighting for shares of a shrinking pot. Killing off the bureaucracy would save money, but somehow I doubt that will be on the agenda.